Adding CI Tests
If you're developing an application that needs a specific
configuration file, input format, or top-level
ComponentAccumulator
-based script, we encourage you to write
continuous integration tests so that we won't break your workflow.
The main test script lives in
test/test-dumper
,
and is invoked by running test-dumper <mode>
, where <mode>
is a
key that selects the configuration file, the input data file, and the
top level test executable to run the dumper. To add a new mode it
should be sufficient to edit test-dumper
and add a new entry under
CONFIGS
, DATAFILES
, and TESTS
.
To add the tests to CI, you'll also have to edit
.gitlab-ci.yml
Making inputs for tests#
We keep the input files small (roughly 10 MB) so that tests are quick to run locally. If you need a new input format for your test, you can create a smaller file in several ways.
Sometimes you can download a dataset from the grid, set up Athena, and run
Merge_tf.py --CA --inputAODFile <xAOD-file> --outputAOD_MRGFile <name>.small.pool.root --maxEvents 10
There should be a simpler command, currently broken!
The previous way to do this was with acmd merge-files
, but that is
currently broken, see ATEAM-941.
acmd merge-files <xAOD-file> -o test.pool.root --evts 10
Unfortunately this may not work for some DAODs. In this case, see the section on making test DAODs.
Uploading test files#
Test files are stored in a dedicated repository via Git LFS. After you've confirmed that you can use your reduced test file, create a MR to this repository including your file. Be sure to commit your file using Git LFS.
Using Git LFS
Using Git LFS requires LFS to be installed. Several installation
options are listed here. If you use conda you can
also run simply run conda install git-lfs
.
However you install Git LFS, you need to run the setup command once per user account.
git lfs install
Once run, Git will automatically commit any files named *.root
using LFS.
Making DAODs from AODs#
You can reproduce a DAOD from its source, but unfortunately there are
a few steps which you have to follow carefully. Suppose you'd like a
10 event version of
DAOD_PHYSVAL.410470.e6337_s3681_r13144_p5057.pool.root
. You'll need
to find:
- The input AOD file, and
- The command to produce the DAOD from the AOD
And then use this information to build a new file.
Central derivation documentation page
The main documentation page on running derivations can be found here.
Finding the input AOD with rucio#
You can dig up the input file with some detective work and rucio. Running
setupATLAS
lsetup rucio
will set up rucio. You can then search with rucio ls
, using the
wildcard *
to fill in unknown parts of the dataset:
rucio ls 'mc20_13TeV:mc20_13TeV*.410470.*e6337_s3681_r13144_p5057*'
will return several datasets (note you have to guess the "scope", it's
usually either mc20_13TeV:
or mc16_13TeV:
). The above gives one
dataset called
mc20_13TeV:mc20_13TeV.410470.PhPy8EG_A14_ttbar_hdamp258p75_nonallhad.deriv.DAOD_PHYSVAL.e6337_s3681_r13144_p5057
It probably came from an input with a similar name, without the
p5057
. Let's try
rucio ls 'mc20_13TeV:mc20_13TeV*.410470.*e6337_s3681_r13144/'
note the /
at the end indicating the end of the container name. That
returns a file named
mc20_13TeV:mc20_13TeV.410470.PhPy8EG_A14_ttbar_hdamp258p75_nonallhad.recon.AOD.e6337_s3681_r13144
Download one of them with rucio get --nrandom 1 <the file>
.
Finding the command to reproduce the derivation#
It's important that you set up the same release and produce the
derivation with the same parameters that it was initially built
with. You can look up the p5057
tag with ami. This will show
you that the cacheName
was 22.0.62
.
You'll need to open a clean shell and set up this release
setupATLAS
asetup Athena,22.0.62
and then produce a new DAOD with
Derivation_tf.py --CA True --AMIConfig p5057 --inputAODFile AOD.pool.root --outputDAODFile small.pool.root --formats PHYSVAL --maxEvents 10
See the central derivation documentation page for up to date instructions
The main documentation page on running derivations can be found here.
Where AOD.pool.root
is the name of the AOD you just
downloaded. There are several additional arguments:
- --AMIConfig p5057
configures the job to use the exact p-tag of choice for the output DAOD.
-
--maxEvents 10
tells the job to stop after 10 events. -
--outputDAODFile small.pool.root
specifies the output type as DAOD and gives a file extension. -
--formats PHYSVAL
specifies that the output is in thePHYSVAL
format.
This should produce a file called DAOD_PHYSVAL.small.pool.root
. You can check the contents with
checkFile DAOD_PHYSVAL.small.pool.root
You can cross check this with the officially produced samples to make sure they contain the same information.