Justin Ngo authored May 16, 2025 and Eric Kunze committed May 19, 2025

Unit test test suite for PyTorch utilizing FxImporter to lower PyTorch
graphs to Torch-MLIR then to TOSA IR. Current infrastructure is mostly
based on the existing TF/TFL frameworks test suite.

This is the initial work for the Torch unit test suite. The end goal
is to add more Torch unit tests for a more robust test suite.

Signed-off-by: Justin Ngo <justin.ngo@arm.com>
Change-Id: I829ddf708556488166bc2bfdd4b52a1cfb4fc34d

- update the version of tools and packages
- add high-level featue description
- create verif/README.md to introduce detailed CTS information

Change-Id: Id80810a700ddf8040954857813cce7464700f45b
Signed-off-by: TatWai Chong <tatwai.chong@arm.com>

Eirene Vlassi Pandi authored Apr 02, 2025 and Eric Kunze committed May 02, 2025

- Added documentation on load_npy and save_npy behavior.
- Included usage examples showing support for both string and DType
 inputs.

Signed-off-by: Eirene Vlassi Pandi <eirene.vlassipandi@arm.com>
Change-Id: I84ec2fd385109857633e8da3eaddea03acab9d16

Make tosa_verif_* tools assume they are in the ref_model repo with a
local build directory.
Update README with profile info, and updated commands to match
assumption of local build directory.
Add pytests for model file locations util.
Fix paths for non-build items with build_sub_dirs.

Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
Change-Id: I70cbcb64423b6782650166507016500268a2496d

Remove exptected_return_code & expected_result_files from desc.json
Clean up readme about desc.json
Add "terminate_early" to desc.json schema and add override warning

Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
Change-Id: Ib7f4c5365ccf58b3c6d1c7e11c4af6817931e6a2

Michael Platings authored Apr 11, 2025 and Jeremy Johnson committed Apr 16, 2025

Updated to use new flatc build instruction and location

Signed-off-by: Michael Platings <michael.platings@arm.com>
Change-Id: I2958db32fec4070508dc61ee85decffee586be6f

Michael Platings authored Feb 20, 2025 and Eric Kunze committed Feb 20, 2025

Previously the documentation implied that a release build would be
created by default. The default is toolchain-specific, hence the
recommendation in the CMake documentation:
"Users should always explicitly specify the build type"

Signed-off-by: Michael Platings <michael.platings@arm.com>
Change-Id: I0b5b793600a548f7b7dd540d588f212f78aae5af

Eirene Vlassi Pandi authored Feb 13, 2025 and Eric Kunze committed Feb 19, 2025

Remove instructions related to handling fp8 from README as
this no longer an issue.

Signed-off-by: Eirene Vlassi Pandi <eirene.vlassipandi@arm.com>
Change-Id: I4a35133ea8712177f4c321edf42e875674d3095a

Won Jong Jeon authored Nov 04, 2024 and Won Jong Jeon committed Nov 06, 2024

Signed-off-by: Won Jeon <won.jeon@arm.com>
Change-Id: I50b67bcc338cbc9ba5a11e95ecb96fa94f9a3105

Victor Anaehobi authored Sep 16, 2024 and Eric Kunze committed Sep 25, 2024

Add pre-commit checks section

Change-Id: I3c613bd25428e1fc0e5e8c0ca0906f20c40aac8b

Peng Sun authored Sep 10, 2024 and Eric Kunze committed Sep 12, 2024

  * add cmd boolean option: --terminate_early
  * add JSON test descriptor field: terminate_early

Signed-off-by: Peng Sun <peng.sun@arm.com>
Change-Id: If3f09cdbf66eb450aaf494ed0d6b0dab2a2d9723

Kaushik Varadharajan authored Jun 26, 2024 and Eric Kunze committed Jul 31, 2024

Installable with `pip install .` in the repository's root directory.

The Python package is built as an .so, which setuptools copies into
py_package/ and installs.

Signed-off-by: Kaushik Varadharajan <kaushik.varadharajan@arm.com>
Change-Id: I20030c02bab6ca0357b55422fc7066ac3a6f6546

Justin Ngo authored Jul 16, 2024 and Eric Kunze committed Jul 16, 2024

- Update cmake requirement in README
- Add cmake version check around CMP0135 policy flag

Signed-off-by: Justin Ngo <justin.ngo@arm.com>
Change-Id: I6d575ad484c5ad95ef6a119661074845b10e7936

Jeremy Johnson authored May 01, 2024 and Eric Kunze committed May 13, 2024

Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
Change-Id: I651ef963b6c14da58dc4cecd8bf4da8fe1a38417

Jeremy Johnson authored Mar 27, 2024 and Eric Kunze committed Mar 28, 2024

Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
Change-Id: I8cd95ed0ae3260eea785e4e0275961534cd1f0c5

Add --test-version option to tosa_verif_conformance_generator to
select the version for tests to output.
Add --output-type to allow json, binary or both files to be created
during conformance generation.
Fix passing schema_path to test runner.
Add explicit verify lib path arg to test runner.

Change-Id: I5f1ad137d713fca408a98470ea77bddf8916c5f3
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>

Jeremy Johnson authored Sep 04, 2023 and Eric Kunze committed Sep 07, 2023

Rename scripts utlities to tosa_* such as tosa_json2numpy

Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
Change-Id: Ie8d584e2afb189fb74cf96b39590c7c27444ba14

Jeremy Johnson authored Aug 02, 2023 and Eric Kunze committed Aug 07, 2023

* Add deprecation notice for conformance_tests repo
* Add deprecation notice for framework tests in conformance

Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
Change-Id: I92b17f4d6d993e47f30d7996a18fc13b934c4cdf

Signed-off-by: Jerry Ge <jerry.ge@arm.com>
Change-Id: I7015de6989940a535a12b737432e5efa8304d9a3

Brings up to date with commonly tested configuration

Change-Id: Id1791f506e6d822d69ee962a9403a7092afeaaf6
Signed-off-by: Eric Kunze <eric.kunze@arm.com>

"tosa_run_tests" -> "tosa_verif_run_tests"

Signed-off-by: Luke Hutton <luke.hutton@arm.com>
Change-Id: Iff8c4874af1dc3e203e1f49563cfa0fd1f373811

* Added IModelRunner interface using pimpl idiom, which allows a user to
  initialize, configure and run the model.
* Added unit tests for IModelRunner.
* Added doctest as third-party submodule.
* Added user options to specify paths for dependencies.
* Moved general func_config functions to separate utility, which removes
  cxxopts dependency.

Signed-off-by: Matthew Sloyan <matthew.sloyan@arm.com>
Change-Id: If42f1f82cd6dadf18911a48dcd5fa579b719aff2

Change-Id: Ia681aa0eb1d42fa31a4658badb376f7b32b2e338
Signed-off-by: Eric Kunze <eric.kunze@arm.com>

Describes the proper way to build debug.

Change-Id: Ie2f888fc71f84af5382e7e8107eb2d3d93a3b056
Signed-off-by: Eric Kunze <eric.kunze@arm.com>

Change-Id: I5cb16ea1f47ee454f03a1d5182827e3fd9f49128
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>

And fixes in tosa_verif_run_tests:
* support for no-color printing
* stop double printing of error messages on verbose
* differentiate result code pass from results check

Change-Id: I26e957013a8d18f7d3d3691067dfb778008a1eea
Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>

Script converts framework and reference model unit tests into a
suitable format for including into a conformance suite.

Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
Change-Id: Ida1ec8a0a7ea31fd3a3f62c4cb52d7cc2bf0b439

Rename to tosa_verif_run_tests to match build_tests
Improve output and system under test support
Improve xunit support
Add results checker
Add utilities json2numpy and json2fbbin
Add set of python tests
Update README.md

Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
Change-Id: Ia09f8e6fd126579b3ba1c1cda95c1326802417ca

Split generator and runner scripts
Add package setup
Add py-dev-env.sh/.bash to allow editing source files during dev
Update README.md with installation info

Signed-off-by: Jeremy Johnson <jeremy.johnson@arm.com>
Change-Id: I172fe426d99e2e9aeeacedc8b8f3b6a79c8bd39d

- Updated example tests, added missing npy files
- Updated README file with fields in desc.json

Change-Id: I5651ed5260866a8381adf620fb6dd6405cb2833b

Change-Id: Iac786eff96183938d2fd11cde9313c6e8e1270a5

- Remove Usage and Format
- Run black on verif/*.py scripts

Signed-off-by: Kevin Cheng <kevin.cheng@arm.com>
Change-Id: Ie81515891eb0039540f614894f4b6b0e0e78ba74

Change-Id: I2f8e7fa63e2ae40203e57d2cc8814bde3b312cb6
Signed-off-by: Eric Kunze <eric.kunze@arm.com>