Mario1159/verilator
1
0
Fork 0
forked from github/verilator
Code Pull Requests Releases 1 Activity

Move test driver documentation into internals.rst

Browse Source
This commit is contained in:
Wilson Snyder 2023-01-21 16:17:26 -05:00
parent 30fd49e21f
commit 3a8288b0f6
3 changed files with 528 additions and 594 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

707
docs/internals.rst
View File

File diff suppressed because it is too large Load Diff

412
test_regress/driver.pl
View File

@ -2664,419 +2664,13 @@ driver.pl - Run regression tests
=head1 SYNOPSIS
driver.pl
make test # In Verilator directory
./driver.pl # Or, to run directly
=head1 DESCRIPTION
driver.pl invokes Verilator or another simulator on each test file.
The driver reports the number of tests which pass, fail, or skipped (some
resource required by the test is not available, such as SystemC).
There are hundreds of tests, and for faster completion you may want to run
the regression tests with OBJCACHE enabled and in parallel on a machine
with many cores. See the -j option and OBJCACHE environment variable.
=head1 TEST CONFIGURATION
The test configuration script (e.g. C<test_regres/t/t_EXAMPLE.pl>) controls
how the test will run by driver.pl. In general it includes a call to the
C<compile> subroutine to compile the test with Verilator (or an alternative
simulator), followed by a call to the C<execute> subroutine to run the
test. Compile-only tests omit the call to C<execute>.
If those complete, the script calls C<ok(1)> to increment the count of
successful tests and then returns 1 as its result.
Both C<compile> and C<execute> take an optional argument hash table to
control their behavior. For example:
compile(
verilator_flags2 => ["--lint-only"],
fails => 1,
);
indicates that when compiling this test, the C<--lint-only> flag should be
passed and that the test is expected to fail.
The full list of arguments can be found by looking at the C<driver.pl>
source code. Some commonly used arguments are:
=over 4
=item all_run_flags
A list of flags to be passed when running the simulator (Verilated model or
one of the other simulators).
=item check_finished
Set to 1 to indicate successful completion of the test is indicated by the
string C<*-* All Finished *-*> being printed on standard output. This is
the normal way for successful tests to finish.
=item expect
A quoted list of strings or regular expression to be matched in the
output. See </HINTS ON WRITING TESTS> for more detail on how this argument
should be used.
=item fails
Set to 1 to indicate this step (C<compile> or C<execute> or C<lint>) is
expected to fail. Tests that are expected to fail generally have _bad in
their filename.
=item make_main
Set to 0 to disable the automatic creation of a C++ test wrapper (for
example when a hand-written test wrapper is provided using C<--exe>).
=item make_top_shell
Set to 0 to disable the automatic creation of a top level shell to run the
executable (for example when a hand-written test wrapper is provided using
C<--exe>).
=item ms_flags
=item ms_flags2
=item ms_run_flags
The equivalent of C<v_flags>, C<v_flags2> and C<all_run_flags>, but only
for use with the ModelSim simulator.
=item nc_flags
=item nc_flags2
=item nc_run_flags
The equivalent of C<v_flags>, C<v_flags2> and C<all_run_flags>, but only
for use with the Cadence NC simulator.
=item iv_flags
=item iv_flags2
=item iv_run_flags
The equivalent of C<v_flags>, C<v_flags2> and C<all_run_flags>, but only
for use with the Icarus Verilog simulator.
=item v_flags
A list of standard Verilog simulator flags to be passed to the simulator
compiler (Verilator or one of the other simulators). This list is create
by the driver and rarely changed, use v_flags2 instead.
=item v_flags2
A list of standard Verilog simulator flags to be passed to the simulator
compiler (Verilator or one of the other simulators). Unlike v_flags, these
options may be overridden in some simulation files.
Similar sets of flags exist for atsim, GHDL, Cadence NC, Modelsim and
Synopsys VCS.
=item vcs_flags
=item vcs_flags2
=item vcs_run_flags
The equivalent of C<v_flags>, C<v_flags2> and C<all_run_flags>, but only
for use with the Synopsys VCS simulator.
=item verilator_flags
=item verilator_flags2
The equivalent of C<v_flags> and C<v_flags2>, but only for use with
Verilator. If a flag is a standard flag (+incdir for example) v_flags2
should be used instead.
=item benchmarksim
Output the number of model evaluations and execution time of a test to
I<test_output_dir>/I<test_name>_benchmarksim.csv. Multiple invocations
of the same test file will append to to the same .csv file.
=item xsim_flags
=item xsim_flags2
=item xsim_run_flags
The equivalent of C<v_flags>, C<v_flags2> and C<all_run_flags>, but only
for use with the Xilinx XSim simulator.
=back
=head2 HINTS ON WRITING TESTS
There is generally no need for the test to create its own main program or
top level shell as the driver creates one automatically, however some tests
require their own C++ or SystemC test harness. This is commonly given the
same name as the test, but with .cpp as suffix
(C<test_regress/t/t_EXAMPLE.cpp>). This can be specified as follows:
compile(
make_top_shell => 0,
make_main => 0,
verilator_flags2 => ["--exe $Self->{t_dir}/$Self->{name}.cpp"], );
Tests should be self-checking, rather than producing lots of output. If a
test succeeds it should print C<*-* All Finished *-*> to standard output
and terminate (in Verilog C<$finish>), if not it should just stop (in
Verilog C<$stop>) as that signals an error.
If termination should be triggered from the C++ wrapper, the following code
can be used:
vl_fatal(__FILE__, __LINE__, "dut", "<error message goes here>");
exit(1);
This can be particularly useful if checking that the Verilator model has
not unexpectedly terminated.
if (contextp->gotFinish()) {
vl_fatal(__FILE__, __LINE__, "dut", "<error message goes here>");
exit(1);
}
Where it might be useful for a test to produce output, it should qualify
this with C<TEST_VERBOSE>. For example in Verilog:
`ifdef TEST_VERBOSE
$write("Conditional generate if MASK [%1d] = %d\n", g, MASK[g]);
`endif
Or in a hand-written C++ wrapper:
#ifdef TEST_VERBOSE
cout << "Read a = " << a << endl;
#endif
The C<expect_filename> specifies a filename that should be used to check
the output results. This should not generally be used to decide if a test
has succeeded. However, in the case of tests that are designed to fail at
compile time, it is the only option. For example:
compile(
fails => 1,
expect_filename => $Self->{golden_filename},
);
Note expect_filename strips some debugging information from the logfile
when comparing.
The C<expect> argument specifies a regular expression which must match the
output.
=head1 DRIVER ARGUMENTS
=over 4
=item --benchmark [<cycles>]
Show execution times of each step. If an optional number is given,
specifies the number of simulation cycles (for tests that support it).
=item --debug
Same as C<verilator --debug>: Use the debug version of Verilator which
enables additional assertions, debugging messages, and structure dump
files.
=item --debugi(-<srcfile>) <level>
Same as C<verilator --debugi level>: Set Verilator internal debugging level
globally to the specified debug level (1-10) or set the specified source
file to the specified level. Higher levels produce more detailed messages
(plain C<--debug> is equivalent to C<--debugi 4>).
=item --dump-tree
Same as C<verilator --dump-tree>: Enable Verilator writing .tree debug
files with dumping level 3, which dumps the standard critical stages. For
details on the format see the Verilator Internals manual.
=item --gdb
Same as C<verilator --gdb>: Run Verilator under the debugger.
=item --gdbbt
Same as C<verilator --gdbbt>: Run Verilator under the debugger, only to
print backtrace information. Requires --debug.
=item --gdbsim
Run Verilator generated executable under the debugger.
=item --golden
Update golden files, equivalent to setting HARNESS_UPDATE_GOLDEN=1.
=item --hashset I<set>/I<numsets>
Split tests based on a hash of the test names into I<numsets> and run only
tests in set number I<set> (0..I<numsets>-1).
=item --help
Displays this message and program version and exits.
=item --j #
Run number of parallel tests, or 0 to determine the count based on the
number of cores installed. Requires Perl's Parallel::Forker package.
=item --quiet
Suppress all output except for failures and progress messages every 15
seconds. Intended for use only in automated regressions. See also
C<--rerun>, and C<--verbose> which is not the opposite of C<--quiet>.
=item --rerun
Rerun all tests that failed in this run. Reruns force the flags
C<--no-quiet --j 1>.
=item --rr
Same as C<verilator --rr>: Run Verilator and record with rr.
=item --rrsim
Run Verilator generated executable and record with rr.
=item --sanitize
Enable address sanitizer to compile Verilated C++ code.
This may detect misuses of memory, such as out-of-bound accesses, use-after-free,
and memory leaks.
=item --site
Run site specific tests also.
=item --stop
Stop on the first error.
=item --trace
Set the simulator specific flags to request waveform tracing.
=item --verbose
Compile and run the test in verbose mode. This means C<TEST_VERBOSE> will
be defined for the test (Verilog and any C++/SystemC wrapper).
=back
=head1 SCENARIO ARGUMENTS
The following options control which simulator is used, and which tests are
run. Multiple flags may be used to run multiple simulators/scenarios
simultaneously.
=over 4
=item --atsim
Run ATSIM simulator tests.
=item --dist
Run simulator-agnostic distribution tests.
=item --ghdl
Run GHDL simulator tests.
=item --iv
Run Icarus Verilog simulator tests.
=item --ms
Run ModelSim simulator tests.
=item --nc
Run Cadence NC-Verilog simulator tests.
=item --vcs
Run Synopsys VCS simulator tests.
=item --vlt
Run Verilator tests in single-threaded mode. Default unless another scenario flag is provided.
=item --vltmt
Run Verilator tests in multithreaded mode.
=item --xsim
Run Xilinx XSim simulator tests.
=back
=head1 ENVIRONMENT
=over 4
=item SYSTEMC
Root directory name of SystemC kit. Only used if SYSTEMC_INCLUDE not set.
=item SYSTEMC_INCLUDE
Directory name with systemc.h in it.
=item VERILATOR_GHDL
Command to use to invoke GHDL.
=item VERILATOR_IVERILOG
Command to use to invoke Icarus Verilog.
=item VERILATOR_MAKE
Command to use to rebuild Verilator and run single test. Defaults to "make &&".
=item VERILATOR_MODELSIM
Command to use to invoke ModelSim.
=item VERILATOR_NCVERILOG
Command to use to invoke ncverilog.
=item VERILATOR_TESTS_SITE
Used with --site, a colon-separated list of directories with tests to be added to testlist.
=item VERILATOR_VCS
Command to use to invoke VCS.
=item VERILATOR_XELAB
Command to use to invoke XSim xelab
=item VERILATOR_XVLOG
Command to use to invoke XSim xvlog
=back
See docs/internals.rst in the distribution for more information.
=head1 DISTRIBUTION

3
test_regress/t/t_driver_random.pl
View File

@ -14,7 +14,8 @@ use Time::HiRes;
scenarios(dist => 1);
if (!$ENV{VERILATOR_TEST_RANDOM_FAILURE}) {
ok("Test is for harness checking only, setenv VERILATOR_TEST_RANDOM_FAILURE=1 ");
print("Test is for harness checking only, setenv VERILATOR_TEST_RANDOM_FAILURE=1\n");
ok(1);
} else {
# Randomly fail to test driver.pl
my ($ign, $t) = Time::HiRes::gettimeofday();