Travis: Use workspaces and per job persistent ccache (#2399)
Change the Travis builds to use workspaces and persistent ccache
We proceed in 2 stages (as before, but using workspaces for
persistence):
1. In the 'build' stage, we clone the repo, build it and
save the whole checkout ($TRAVIS_BUILD_DIR) as a workspace
2. In the 'test' stage, rather than cloning the repo, multiple jobs
pull down the same workspace we built to run the tests from
This enables:
- Reuse of the build in multiple test jobs (this is what we used the Travis
cache for before)
- Each job having a separate persistent Travis cache, which now only
contains the ccache. This means all jobs, including 'build' and 'test'
jobs can make maximum use of ccache across runs. This drastically cuts
down build times when the ccache hits, which is very often the case for
'test' jobs. Also, the separate caches only store the objects build by
the particular job that owns the cache, so we can keep the per job
ccache small.
If the commit message contains '[travis ccache clear]', the ccache will
be cleared at the beginning of the build. This can be used to test build
complete within the 50 minute timeout imposed by Travis, even without a
persistent ccache.
2020-06-03 20:10:13 +00:00
|
|
|
#!/usr/bin/env bash
|
2020-12-04 15:30:46 +00:00
|
|
|
# DESCRIPTION: Verilator: CI main job script
|
Travis: Use workspaces and per job persistent ccache (#2399)
Change the Travis builds to use workspaces and persistent ccache
We proceed in 2 stages (as before, but using workspaces for
persistence):
1. In the 'build' stage, we clone the repo, build it and
save the whole checkout ($TRAVIS_BUILD_DIR) as a workspace
2. In the 'test' stage, rather than cloning the repo, multiple jobs
pull down the same workspace we built to run the tests from
This enables:
- Reuse of the build in multiple test jobs (this is what we used the Travis
cache for before)
- Each job having a separate persistent Travis cache, which now only
contains the ccache. This means all jobs, including 'build' and 'test'
jobs can make maximum use of ccache across runs. This drastically cuts
down build times when the ccache hits, which is very often the case for
'test' jobs. Also, the separate caches only store the objects build by
the particular job that owns the cache, so we can keep the per job
ccache small.
If the commit message contains '[travis ccache clear]', the ccache will
be cleared at the beginning of the build. This can be used to test build
complete within the 50 minute timeout imposed by Travis, even without a
persistent ccache.
2020-06-03 20:10:13 +00:00
|
|
|
#
|
|
|
|
|
# Copyright 2020 by Geza Lore. This program is free software; you
|
|
|
|
|
# can redistribute it and/or modify it under the terms of either the GNU
|
|
|
|
|
# Lesser General Public License Version 3 or the Perl Artistic License
|
|
|
|
|
# Version 2.0.
|
|
|
|
|
#
|
|
|
|
|
# SPDX-License-Identifier: LGPL-3.0-only OR Artistic-2.0
|
|
|
|
|
|
|
|
|
|
################################################################################
|
|
|
|
|
# This is the main script executed in the 'script' phase by all jobs. We use a
|
2020-12-04 15:30:46 +00:00
|
|
|
# single script to keep the CI setting simple. We pass job parameters via
|
|
|
|
|
# environment variables using 'env' keys.
|
Travis: Use workspaces and per job persistent ccache (#2399)
Change the Travis builds to use workspaces and persistent ccache
We proceed in 2 stages (as before, but using workspaces for
persistence):
1. In the 'build' stage, we clone the repo, build it and
save the whole checkout ($TRAVIS_BUILD_DIR) as a workspace
2. In the 'test' stage, rather than cloning the repo, multiple jobs
pull down the same workspace we built to run the tests from
This enables:
- Reuse of the build in multiple test jobs (this is what we used the Travis
cache for before)
- Each job having a separate persistent Travis cache, which now only
contains the ccache. This means all jobs, including 'build' and 'test'
jobs can make maximum use of ccache across runs. This drastically cuts
down build times when the ccache hits, which is very often the case for
'test' jobs. Also, the separate caches only store the objects build by
the particular job that owns the cache, so we can keep the per job
ccache small.
If the commit message contains '[travis ccache clear]', the ccache will
be cleared at the beginning of the build. This can be used to test build
complete within the 50 minute timeout imposed by Travis, even without a
persistent ccache.
2020-06-03 20:10:13 +00:00
|
|
|
################################################################################
|
|
|
|
|
|
|
|
|
|
set -e
|
|
|
|
|
set -x
|
|
|
|
|
|
|
|
|
|
fatal() {
|
|
|
|
|
echo "ERROR: $(basename "$0"): $1" >&2; exit 1;
|
|
|
|
|
}
|
|
|
|
|
|
2021-06-13 19:38:24 +00:00
|
|
|
if [ "$CI_M32" = "0" ]; then
|
|
|
|
|
unset CI_M32
|
|
|
|
|
elif [ "$CI_M32" != "1" ]; then
|
|
|
|
|
fatal "\$CI_M32 must be '0' or '1'";
|
|
|
|
|
fi
|
|
|
|
|
|
2020-12-04 15:30:46 +00:00
|
|
|
if [ "$CI_OS_NAME" = "linux" ]; then
|
2020-06-04 20:08:32 +00:00
|
|
|
export MAKE=make
|
2020-06-22 09:13:54 +00:00
|
|
|
NPROC=$(nproc)
|
2020-12-04 15:30:46 +00:00
|
|
|
elif [ "$CI_OS_NAME" = "osx" ]; then
|
2020-06-04 20:08:32 +00:00
|
|
|
export MAKE=make
|
2020-06-22 09:13:54 +00:00
|
|
|
NPROC=$(sysctl -n hw.logicalcpu)
|
2020-12-04 15:30:46 +00:00
|
|
|
elif [ "$CI_OS_NAME" = "freebsd" ]; then
|
2020-06-04 20:08:32 +00:00
|
|
|
export MAKE=gmake
|
|
|
|
|
NPROC=$(sysctl -n hw.ncpu)
|
2020-06-22 09:13:54 +00:00
|
|
|
else
|
2020-12-04 15:30:46 +00:00
|
|
|
fatal "Unknown os: '$CI_OS_NAME'"
|
2020-06-22 09:13:54 +00:00
|
|
|
fi
|
|
|
|
|
|
2020-12-04 15:30:46 +00:00
|
|
|
if [ "$CI_BUILD_STAGE_NAME" = "build" ]; then
|
Travis: Use workspaces and per job persistent ccache (#2399)
Change the Travis builds to use workspaces and persistent ccache
We proceed in 2 stages (as before, but using workspaces for
persistence):
1. In the 'build' stage, we clone the repo, build it and
save the whole checkout ($TRAVIS_BUILD_DIR) as a workspace
2. In the 'test' stage, rather than cloning the repo, multiple jobs
pull down the same workspace we built to run the tests from
This enables:
- Reuse of the build in multiple test jobs (this is what we used the Travis
cache for before)
- Each job having a separate persistent Travis cache, which now only
contains the ccache. This means all jobs, including 'build' and 'test'
jobs can make maximum use of ccache across runs. This drastically cuts
down build times when the ccache hits, which is very often the case for
'test' jobs. Also, the separate caches only store the objects build by
the particular job that owns the cache, so we can keep the per job
ccache small.
If the commit message contains '[travis ccache clear]', the ccache will
be cleared at the beginning of the build. This can be used to test build
complete within the 50 minute timeout imposed by Travis, even without a
persistent ccache.
2020-06-03 20:10:13 +00:00
|
|
|
##############################################################################
|
|
|
|
|
# Build verilator
|
|
|
|
|
|
2020-06-22 09:13:54 +00:00
|
|
|
if [ "$COVERAGE" != 1 ]; then
|
|
|
|
|
autoconf
|
2021-06-13 19:38:24 +00:00
|
|
|
./configure --enable-longtests --enable-ccwarn ${CI_M32:+--enable-m32}
|
2021-06-13 16:32:49 +00:00
|
|
|
ccache -z
|
2020-08-21 12:31:27 +00:00
|
|
|
"$MAKE" -j "$NPROC" -k
|
2021-06-13 16:32:49 +00:00
|
|
|
ccache -s
|
2020-12-04 15:30:46 +00:00
|
|
|
if [ "$CI_OS_NAME" = "osx" ]; then
|
2020-06-25 13:12:09 +00:00
|
|
|
file bin/verilator_bin
|
|
|
|
|
file bin/verilator_bin_dbg
|
|
|
|
|
md5 bin/verilator_bin
|
|
|
|
|
md5 bin/verilator_bin_dbg
|
|
|
|
|
stat bin/verilator_bin
|
|
|
|
|
stat bin/verilator_bin_dbg
|
|
|
|
|
fi
|
Travis: Use workspaces and per job persistent ccache (#2399)
Change the Travis builds to use workspaces and persistent ccache
We proceed in 2 stages (as before, but using workspaces for
persistence):
1. In the 'build' stage, we clone the repo, build it and
save the whole checkout ($TRAVIS_BUILD_DIR) as a workspace
2. In the 'test' stage, rather than cloning the repo, multiple jobs
pull down the same workspace we built to run the tests from
This enables:
- Reuse of the build in multiple test jobs (this is what we used the Travis
cache for before)
- Each job having a separate persistent Travis cache, which now only
contains the ccache. This means all jobs, including 'build' and 'test'
jobs can make maximum use of ccache across runs. This drastically cuts
down build times when the ccache hits, which is very often the case for
'test' jobs. Also, the separate caches only store the objects build by
the particular job that owns the cache, so we can keep the per job
ccache small.
If the commit message contains '[travis ccache clear]', the ccache will
be cleared at the beginning of the build. This can be used to test build
complete within the 50 minute timeout imposed by Travis, even without a
persistent ccache.
2020-06-03 20:10:13 +00:00
|
|
|
else
|
2021-01-11 00:15:39 +00:00
|
|
|
nodist/code_coverage --stages 0-2
|
Travis: Use workspaces and per job persistent ccache (#2399)
Change the Travis builds to use workspaces and persistent ccache
We proceed in 2 stages (as before, but using workspaces for
persistence):
1. In the 'build' stage, we clone the repo, build it and
save the whole checkout ($TRAVIS_BUILD_DIR) as a workspace
2. In the 'test' stage, rather than cloning the repo, multiple jobs
pull down the same workspace we built to run the tests from
This enables:
- Reuse of the build in multiple test jobs (this is what we used the Travis
cache for before)
- Each job having a separate persistent Travis cache, which now only
contains the ccache. This means all jobs, including 'build' and 'test'
jobs can make maximum use of ccache across runs. This drastically cuts
down build times when the ccache hits, which is very often the case for
'test' jobs. Also, the separate caches only store the objects build by
the particular job that owns the cache, so we can keep the per job
ccache small.
If the commit message contains '[travis ccache clear]', the ccache will
be cleared at the beginning of the build. This can be used to test build
complete within the 50 minute timeout imposed by Travis, even without a
persistent ccache.
2020-06-03 20:10:13 +00:00
|
|
|
fi
|
2020-12-04 15:30:46 +00:00
|
|
|
elif [ "$CI_BUILD_STAGE_NAME" = "test" ]; then
|
Travis: Use workspaces and per job persistent ccache (#2399)
Change the Travis builds to use workspaces and persistent ccache
We proceed in 2 stages (as before, but using workspaces for
persistence):
1. In the 'build' stage, we clone the repo, build it and
save the whole checkout ($TRAVIS_BUILD_DIR) as a workspace
2. In the 'test' stage, rather than cloning the repo, multiple jobs
pull down the same workspace we built to run the tests from
This enables:
- Reuse of the build in multiple test jobs (this is what we used the Travis
cache for before)
- Each job having a separate persistent Travis cache, which now only
contains the ccache. This means all jobs, including 'build' and 'test'
jobs can make maximum use of ccache across runs. This drastically cuts
down build times when the ccache hits, which is very often the case for
'test' jobs. Also, the separate caches only store the objects build by
the particular job that owns the cache, so we can keep the per job
ccache small.
If the commit message contains '[travis ccache clear]', the ccache will
be cleared at the beginning of the build. This can be used to test build
complete within the 50 minute timeout imposed by Travis, even without a
persistent ccache.
2020-06-03 20:10:13 +00:00
|
|
|
##############################################################################
|
|
|
|
|
# Run tests
|
|
|
|
|
|
2020-12-10 23:12:26 +00:00
|
|
|
export VERILATOR_TEST_NO_CONTRIBUTORS=1 # Separate workflow check
|
|
|
|
|
|
2020-12-04 15:30:46 +00:00
|
|
|
if [ "$CI_OS_NAME" = "osx" ]; then
|
2020-12-10 23:12:26 +00:00
|
|
|
export VERILATOR_TEST_NO_GDB=1 # Pain to get GDB to work on OS X
|
2023-01-08 23:17:24 +00:00
|
|
|
# TODO below may no longer be required as configure checks for -pg
|
2020-12-10 23:12:26 +00:00
|
|
|
export VERILATOR_TEST_NO_GPROF=1 # Apple Clang has no -pg
|
2020-06-22 09:13:54 +00:00
|
|
|
# export PATH="/Applications/gtkwave.app/Contents/Resources/bin:$PATH" # fst2vcd
|
2020-06-25 13:12:09 +00:00
|
|
|
file bin/verilator_bin
|
|
|
|
|
file bin/verilator_bin_dbg
|
|
|
|
|
md5 bin/verilator_bin
|
|
|
|
|
md5 bin/verilator_bin_dbg
|
|
|
|
|
stat bin/verilator_bin
|
|
|
|
|
stat bin/verilator_bin_dbg
|
|
|
|
|
# For some reason, the dbg exe is corrupted by this point ('file' reports
|
|
|
|
|
# it as data rather than a Mach-O). Unclear if this is an OS X issue or
|
2020-12-10 14:06:01 +00:00
|
|
|
# CI's. Remove the file and re-link...
|
2020-06-25 13:12:09 +00:00
|
|
|
rm bin/verilator_bin_dbg
|
2020-08-21 12:31:27 +00:00
|
|
|
"$MAKE" -j "$NPROC" -k
|
2020-12-04 15:30:46 +00:00
|
|
|
elif [ "$CI_OS_NAME" = "freebsd" ]; then
|
2020-06-04 20:08:32 +00:00
|
|
|
export VERILATOR_TEST_NO_GDB=1 # Disable for now, ideally should run
|
2023-01-08 23:17:24 +00:00
|
|
|
# TODO below may no longer be required as configure checks for -pg
|
2020-06-04 20:08:32 +00:00
|
|
|
export VERILATOR_TEST_NO_GPROF=1 # gprof is a bit different on FreeBSD, disable
|
2020-06-22 09:13:54 +00:00
|
|
|
fi
|
|
|
|
|
|
2020-12-23 17:53:05 +00:00
|
|
|
# Run sanitize on Ubuntu 20.04 only
|
2021-06-13 19:38:24 +00:00
|
|
|
[ "$CI_RUNS_ON" = 'ubuntu-20.04' ] && [ "$CI_M32" = "" ] && sanitize='--sanitize' || sanitize=''
|
2020-12-23 17:53:05 +00:00
|
|
|
|
2020-06-22 09:13:54 +00:00
|
|
|
# Run the specified test
|
2021-06-13 16:32:49 +00:00
|
|
|
ccache -z
|
2020-06-22 09:13:54 +00:00
|
|
|
case $TESTS in
|
|
|
|
|
dist-vlt-0)
|
2021-06-09 22:51:03 +00:00
|
|
|
"$MAKE" -C test_regress SCENARIOS="--dist --vlt $sanitize" DRIVER_HASHSET=--hashset=0/3
|
2020-06-22 09:13:54 +00:00
|
|
|
;;
|
|
|
|
|
dist-vlt-1)
|
2021-06-09 22:51:03 +00:00
|
|
|
"$MAKE" -C test_regress SCENARIOS="--dist --vlt $sanitize" DRIVER_HASHSET=--hashset=1/3
|
|
|
|
|
;;
|
|
|
|
|
dist-vlt-2)
|
|
|
|
|
"$MAKE" -C test_regress SCENARIOS="--dist --vlt $sanitize" DRIVER_HASHSET=--hashset=2/3
|
2020-06-22 09:13:54 +00:00
|
|
|
;;
|
|
|
|
|
vltmt-0)
|
2020-06-04 20:08:32 +00:00
|
|
|
"$MAKE" -C test_regress SCENARIOS=--vltmt DRIVER_HASHSET=--hashset=0/2
|
2020-06-22 09:13:54 +00:00
|
|
|
;;
|
|
|
|
|
vltmt-1)
|
2020-06-04 20:08:32 +00:00
|
|
|
"$MAKE" -C test_regress SCENARIOS=--vltmt DRIVER_HASHSET=--hashset=1/2
|
2020-06-22 09:13:54 +00:00
|
|
|
;;
|
2020-12-08 13:00:24 +00:00
|
|
|
coverage-all)
|
2021-01-11 00:15:39 +00:00
|
|
|
nodist/code_coverage --stages 1-
|
2020-12-08 13:00:24 +00:00
|
|
|
;;
|
2020-06-22 09:13:54 +00:00
|
|
|
coverage-dist)
|
2021-01-11 00:15:39 +00:00
|
|
|
nodist/code_coverage --stages 1- --scenarios=--dist
|
2020-06-22 09:13:54 +00:00
|
|
|
;;
|
|
|
|
|
coverage-vlt-0)
|
2021-03-07 21:41:46 +00:00
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vlt --hashset=0/10
|
2020-06-22 09:13:54 +00:00
|
|
|
;;
|
|
|
|
|
coverage-vlt-1)
|
2021-03-07 21:41:46 +00:00
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vlt --hashset=1/10
|
2020-06-22 09:13:54 +00:00
|
|
|
;;
|
|
|
|
|
coverage-vlt-2)
|
2021-03-07 21:41:46 +00:00
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vlt --hashset=2/10
|
2020-06-22 09:13:54 +00:00
|
|
|
;;
|
|
|
|
|
coverage-vlt-3)
|
2021-03-07 21:41:46 +00:00
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vlt --hashset=3/10
|
|
|
|
|
;;
|
|
|
|
|
coverage-vlt-4)
|
|
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vlt --hashset=4/10
|
|
|
|
|
;;
|
|
|
|
|
coverage-vlt-5)
|
|
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vlt --hashset=5/10
|
|
|
|
|
;;
|
|
|
|
|
coverage-vlt-6)
|
|
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vlt --hashset=6/10
|
|
|
|
|
;;
|
|
|
|
|
coverage-vlt-7)
|
|
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vlt --hashset=7/10
|
|
|
|
|
;;
|
|
|
|
|
coverage-vlt-8)
|
|
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vlt --hashset=8/10
|
|
|
|
|
;;
|
|
|
|
|
coverage-vlt-9)
|
|
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vlt --hashset=9/10
|
2020-06-22 09:13:54 +00:00
|
|
|
;;
|
|
|
|
|
coverage-vltmt-0)
|
2021-03-07 21:41:46 +00:00
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vltmt --hashset=0/10
|
2020-06-22 09:13:54 +00:00
|
|
|
;;
|
|
|
|
|
coverage-vltmt-1)
|
2021-03-07 21:41:46 +00:00
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vltmt --hashset=1/10
|
2020-06-22 09:13:54 +00:00
|
|
|
;;
|
|
|
|
|
coverage-vltmt-2)
|
2021-03-07 21:41:46 +00:00
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vltmt --hashset=2/10
|
2020-06-22 09:13:54 +00:00
|
|
|
;;
|
|
|
|
|
coverage-vltmt-3)
|
2021-03-07 21:41:46 +00:00
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vltmt --hashset=3/10
|
|
|
|
|
;;
|
|
|
|
|
coverage-vltmt-4)
|
|
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vltmt --hashset=4/10
|
|
|
|
|
;;
|
|
|
|
|
coverage-vltmt-5)
|
|
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vltmt --hashset=5/10
|
|
|
|
|
;;
|
|
|
|
|
coverage-vltmt-6)
|
|
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vltmt --hashset=6/10
|
|
|
|
|
;;
|
|
|
|
|
coverage-vltmt-7)
|
|
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vltmt --hashset=7/10
|
|
|
|
|
;;
|
|
|
|
|
coverage-vltmt-8)
|
|
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vltmt --hashset=8/10
|
|
|
|
|
;;
|
|
|
|
|
coverage-vltmt-9)
|
|
|
|
|
nodist/code_coverage --stages 1- --scenarios=--vltmt --hashset=9/10
|
2020-06-22 09:13:54 +00:00
|
|
|
;;
|
|
|
|
|
*)
|
|
|
|
|
fatal "Unknown test: $TESTS"
|
|
|
|
|
;;
|
|
|
|
|
esac
|
2021-06-13 16:32:49 +00:00
|
|
|
ccache -s
|
|
|
|
|
|
2021-04-24 13:38:38 +00:00
|
|
|
# Upload coverage data
|
2020-06-22 09:13:54 +00:00
|
|
|
if [[ $TESTS == coverage-* ]]; then
|
2021-04-24 13:38:38 +00:00
|
|
|
bash <(cat ci/coverage-upload.sh) -f nodist/obj_dir/coverage/app_total.info
|
Travis: Use workspaces and per job persistent ccache (#2399)
Change the Travis builds to use workspaces and persistent ccache
We proceed in 2 stages (as before, but using workspaces for
persistence):
1. In the 'build' stage, we clone the repo, build it and
save the whole checkout ($TRAVIS_BUILD_DIR) as a workspace
2. In the 'test' stage, rather than cloning the repo, multiple jobs
pull down the same workspace we built to run the tests from
This enables:
- Reuse of the build in multiple test jobs (this is what we used the Travis
cache for before)
- Each job having a separate persistent Travis cache, which now only
contains the ccache. This means all jobs, including 'build' and 'test'
jobs can make maximum use of ccache across runs. This drastically cuts
down build times when the ccache hits, which is very often the case for
'test' jobs. Also, the separate caches only store the objects build by
the particular job that owns the cache, so we can keep the per job
ccache small.
If the commit message contains '[travis ccache clear]', the ccache will
be cleared at the beginning of the build. This can be used to test build
complete within the 50 minute timeout imposed by Travis, even without a
persistent ccache.
2020-06-03 20:10:13 +00:00
|
|
|
fi
|
|
|
|
|
else
|
|
|
|
|
##############################################################################
|
|
|
|
|
# Unknown build stage
|
2020-12-04 15:30:46 +00:00
|
|
|
fatal "Unknown build stage: '$CI_BUILD_STAGE_NAME'"
|
Travis: Use workspaces and per job persistent ccache (#2399)
Change the Travis builds to use workspaces and persistent ccache
We proceed in 2 stages (as before, but using workspaces for
persistence):
1. In the 'build' stage, we clone the repo, build it and
save the whole checkout ($TRAVIS_BUILD_DIR) as a workspace
2. In the 'test' stage, rather than cloning the repo, multiple jobs
pull down the same workspace we built to run the tests from
This enables:
- Reuse of the build in multiple test jobs (this is what we used the Travis
cache for before)
- Each job having a separate persistent Travis cache, which now only
contains the ccache. This means all jobs, including 'build' and 'test'
jobs can make maximum use of ccache across runs. This drastically cuts
down build times when the ccache hits, which is very often the case for
'test' jobs. Also, the separate caches only store the objects build by
the particular job that owns the cache, so we can keep the per job
ccache small.
If the commit message contains '[travis ccache clear]', the ccache will
be cleared at the beginning of the build. This can be used to test build
complete within the 50 minute timeout imposed by Travis, even without a
persistent ccache.
2020-06-03 20:10:13 +00:00
|
|
|
fi
|
