Found 582 jobs Not-understood crash indication Binary file /home/teuthworker/archive/mchangir-2020-09-30_15:36:15-fs-wip-mchangir-cephfs-snapshot-replication-distro-basic-smithi/5484327/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/mchangir-2020-09-30_15:36:15-fs-wip-mchangir-cephfs-snapshot-replication-distro-basic-smithi/5484327/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/mchangir-2020-09-30_15:36:15-fs-wip-mchangir-cephfs-snapshot-replication-distro-basic-smithi/5484327/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/mchangir-2020-09-30_15:36:15-fs-wip-mchangir-cephfs-snapshot-replication-distro-basic-smithi/5484327/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/mchangir-2020-09-30_15:36:15-fs-wip-mchangir-cephfs-snapshot-replication-distro-basic-smithi/5484327/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/mchangir-2020-09-30_15:36:15-fs-wip-mchangir-cephfs-snapshot-replication-distro-basic-smithi/5484327/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/mchangir-2020-09-30_15:36:15-fs-wip-mchangir-cephfs-snapshot-replication-distro-basic-smithi/5484327/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/mchangir-2020-09-30_15:36:15-fs-wip-mchangir-cephfs-snapshot-replication-distro-basic-smithi/5484327/teuthology.log matches Not-understood crash indication Not-understood crash indication Binary file /home/teuthworker/archive/mchangir-2020-09-30_15:36:15-fs-wip-mchangir-cephfs-snapshot-replication-distro-basic-smithi/5484327/teuthology.log matches Not-understood crash indication Found 8 distinct failure reasons Failure: Command failed on smithi204 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' 6 jobs: ['5484269', '5484255', '5484305', '5484610', '5484615', '5484578'] suites intersection: ['conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['ceph-thrash/mon', 'clusters/1-mds-1-client-coloc', 'clusters/1-mds-4-client-coloc', 'clusters/fixed-2-ucephfs', 'conf/{client', 'fs/basic_functional/{begin', 'fs/basic_workload/{begin', 'fs/thrash/{begin', 'inline/no', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'no_client_pidfile', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'tasks/cap-flush}', 'tasks/cfuse_workunit_norstats}', 'tasks/cfuse_workunit_snaptests}', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_fsx}', 'tasks/cfuse_workunit_suites_pjd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: {'Failure object was': {'smithi081.front.sepia.ceph.com': {'msg': 'Depsolve Error occured: \\n Problem: cannot install the best candidate for the job\\n - nothing provides libgcc >= 8.3.1-5.el8.0.2 needed by gcc-8.3.1-5.el8.0.2.x86_64\\n - nothing provides libgomp = 8.3.1-5.el8.0.2 needed by gcc-8.3.1-5.el8.0.2.x86_64', 'failures': [], 'results': [], 'rc': 1, 'invocation': {'module_args': {'name': ['dnf-utils', 'sysstat', 'libedit', 'boost-thread', 'xfsprogs', 'gdisk', 'parted', 'libgcrypt', 'fuse-libs', 'openssl', 'libuuid', 'podman', 'podman-docker', 'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump', 'blktrace', 'usbredir', 'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git', 'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img', 'qemu-kvm-core', 'qemu-kvm-block-rbd', 'libacl-devel', 'dbench', 'autoconf', 'gdb', 'iozone'], 'state': 'present', 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_cache': False, 'update_only': False, 'validate_certs': True, 'lock_timeout': 30, 'conf_file': 'None', 'disable_excludes': 'None', 'download_dir': 'None', 'list': 'None', 'releasever': 'None'}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '/')"} 3 jobs: ['5484702', '5484698', '5484455'] suites intersection: ['clusters/fixed-2-ucephfs', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['centos_latest', 'clusters/fixed-2-ucephfs', 'conf/{client', 'fs/basic_workload/{begin', 'fs/permission/{begin', 'fs/verify/{begin', 'inline/no', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore-ec/filestore-xfs', 'omap_limit/10000', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/cfuse_workunit_suites_pjd}', 'validater/lockdep}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) 2 jobs: ['5484252', '5484541'] suites intersection: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'tasks/alternate-pool}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/alternate-pool}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi093 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2c14aad3268ff9d6fb401ad24471b78474a4778b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' ceph version 16.0.0-5811-g2c14aad3 (2c14aad3268ff9d6fb401ad24471b78474a4778b) pacific (dev) 1: () 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f04b898b549] 5: () 6: (Monitor::~Monitor()+0xe62) [0x55b48383e3f2] 7: (Monitor::~Monitor()+0xd) [0x55b48383e46d] 8: main() 9: __libc_start_main() 10: _start() 1 jobs: ['5484700'] suites: ['ceph-thrash/mon', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'fs/thrash/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{rhel_8}', 'tasks/cfuse_workunit_snaptests}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) 2 jobs: ['5484782', '5484493'] suites intersection: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'tasks/test_journal_migration}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'supported-random-distros$/{rhel_8}', 'tasks/test_journal_migration}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test fs/quota/quota.sh) on smithi067 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2c14aad3268ff9d6fb401ad24471b78474a4778b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' 2 jobs: ['5484318', '5484608'] suites intersection: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'tasks/cfuse_workunit_quota}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{rhel_8}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/cfuse_workunit_quota}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test suites/fsstress.sh) on smithi162 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2c14aad3268ff9d6fb401ad24471b78474a4778b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 1 jobs: ['5484327'] suites: ['ceph-thrash/mon', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'fs/thrash/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{ubuntu_latest}', 'tasks/cfuse_workunit_suites_fsstress}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: "2020-09-30T18:32:53.115987+0000 osd.1 (osd.1) 3 : cluster [WRN] slow request osd_op(client.4654.0:40816 4.7s0 4:f2a80dfe:::100000001f8.00000019:head [write 466944~4096 in=4096b,write 1818624~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e26) initiated 2020-09-30T18:29:52.930000+0000 currently started" in cluster log 1 jobs: ['5484366'] suites: ['clusters/fixed-2-ucephfs', 'conf/{client', 'fs/basic_workload/{begin', 'inline/yes', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'tasks/cfuse_workunit_suites_fsync}', 'whitelist_health', 'whitelist_wrongly_marked_down}']