Suhail M authored Apr 24, 2025 and Jakub Sujak committed Apr 24, 2025

- Adds packing and matmul kernels for FP32 SME Indirect GEMM
- Adds tests for Indirect Gemm with FP32 inputs/outputs.

Signed-off-by: Mohammed Suhail Munshi <MohammedSuhail.Munshi@arm.com>

Reviewed-by: Anton Bondarenko <anton.bondarenko@arm.com>
Reviewed-by: Jakub Sujak <jakub.sujak@arm.com>
Reviewed-by: Mohammed Suhail Munshi <mohammedsuhail.munshi@arm.com>
Reviewed-by: Emil Ohlsson <emil.ohlsson@arm.com>
Approved-by: Emil Ohlsson <emil.ohlsson@arm.com>

Jens Elofsson authored Apr 24, 2025 and Jakub Sujak committed Apr 24, 2025

Signed-off-by: Jens Elofsson <jens.elofsson@arm.com>

Approved-by: Anton Bondarenko <anton.bondarenko@arm.com>
Approved-by: Jakub Sujak <jakub.sujak@arm.com>

This change introduces three new kernels
* `kai_lhs_imatmul_pack_x16p2vlx2_x16p_sme`
* `kai_rhs_imatmul_pack_kxn_x16p2vlx2b_x16_x16_sme`
* `kai_imatmul_clamp_f16_f16p2vlx2_f16p2vlx2_2vlx2vl_sme2_mopa`

These are used for indirect matmul, for 16-bit floating point data.

This change also adds unit testing for the FP16 imatmul kernels. The
code is written in a type agnostic manner, as to easily allow testing
for other data types with very low effort. This required the addition of
non-templated `read`/`write` functions, as to allow runtime-generic
access.

Signed-off-by: Emil Ohlsson <emil.ohlsson@arm.com>

Reviewed-by: Anton Bondarenko <anton.bondarenko@arm.com>
Reviewed-by: Jakub Sujak <jakub.sujak@arm.com>
Reviewed-by: Emil Ohlsson <emil.ohlsson@arm.com>
Approved-by: Anton Bondarenko <anton.bondarenko@arm.com>

Evie Wright authored Apr 15, 2025 and Jakub Sujak committed Apr 15, 2025

Signed-off-by: Evie Wright <evie.wright@arm.com>

Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Update all version indicators to 1.7.0.

Signed-off-by: Jens Elofsson <jens.elofsson@arm.com>

Approved-by: Emil Ohlsson <emil.ohlsson@arm.com>

Anitha Raj authored Apr 11, 2025 and Viet-Hoa Do committed Apr 11, 2025

Micro-kernels to compute the matrix multiplication of dynamically quantized asymmetric signed 8-bit integer with per-channel quantization (QAI8DX) LHS matrix and quantized symmetric 4-bit signed integer with per-channel quantization (QSI4CX) RHS matrix and the accumulation of the result into a half-precision (F16):

Matrix multiplication (MxN) Micro-kernels of QAI8DX LHS and QSI4CX RHS with F16 output, optimized for FEAT_I8MM and FEAT_DotProd. 
Matrix multiplication (1xN) Micro-kernels of QAI8DX LHS and QSI4CX RHS with F16 output, optimized for FEAT_DotProd.

Signed-off-by: Anitha Raj <anitha.raj@arm.com>

Signed-off-by: Evie Wright <evie.wright@arm.com>

Approved-by: Viet-Hoa Do <viet-hoa.do@arm.com>

Anitha Raj authored Apr 09, 2025 and Anton Bondarenko committed Apr 09, 2025

Micro-kernels to compute the matrix multiplication of dynamically quantized symmetric signed 8-bit integer with per-block quantization (QSI8D32) LHS matrix and quantized asymmetric 4-bit signed integer with per-block quantization (QAI4C32) RHS matrix and the accumulation of the result into a single-precision (F32) and half-precision (F16) output:

- Matrix multiplication (MxN) Micro-kernels of QSI8D32 LHS and QAI4C32 RHS with F32 output, optimized for FEAT_I8MM.
- Matrix multiplication (1xN) Micro-kernels of QSI8D32 LHS and QAI4C32 RHS with F32 output, optimized for FEAT_DotProd.
- Matrix multiplication (MxN) Micro-kernels of QSI8D32 LHS and QAI4C32 RHS with F16 output, optimized for FEAT_I8MM.
- Matrix multiplication (1xN) Micro-kernels of QSI8D32 LHS and QAI4C32 RHS with F16 output, optimized for FEAT_DotProd.

Signed-off-by: Anitha Raj <anitha.raj@arm.com>

Reviewed-by: Viet-Hoa Do <viet-hoa.do@arm.com>
Reviewed-by: Anitha Raj <anitha.raj@arm.com>
Reviewed-by: Anton Bondarenko <anton.bondarenko@arm.com>
Approved-by: Anton Bondarenko <anton.bondarenko@arm.com>

This change introduces three new kernels:

* kai_imatmul_clamp_qai8_qai8p2vlx4_qsi8cxpsb2vlx4_2vlx2vl_sme2_mopa
* kai_lhs_imatmul_pack_x8p2vlx4_x8p_sme
* kai_rhs_imatmul_pack_kxn_qsi8cxp2vlx4sb_qs8cx_f32_i32_sme

These kernels are used for _indirect matmul_. The big difference between
these kernels and matmul kernels is that the LHS packing kernel takes an
indirection buffer where each pointer refers to a chunk in K dimension.
The pointers are laid out in a packed manner, where instead of being in
row major order, a column of `get_m_step` chunk pointers are placed
linearly in indirection buffer.

In addition to the kernels themselves, the
`matmul_clamp_qai8_qai8p_qsi8cxp_test.cpp` is extended to perform
testing of these new kernels. The testing flow for these new kernels is
a bit different, in that the packing kernels themselves are not directly
tested, instead only end-to-end flow is tested.

Signed-off-by: Emil Ohlsson <emil.ohlsson@arm.com>
Signed-off-by: Felix Thomasmathibalan <felixjohnny.thomasmathibalan@arm.com>
Signed-off-by: Mohammed Suhail Munshi <MohammedSuhail.Munshi@arm.com>

Reviewed-by: Viet-Hoa Do <viet-hoa.do@arm.com>
Reviewed-by: Anton Bondarenko <anton.bondarenko@arm.com>
Reviewed-by: Jakub Sujak <jakub.sujak@arm.com>
Reviewed-by: Emil Ohlsson <emil.ohlsson@arm.com>
Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Update all version indicators to 1.6.0.

Signed-off-by: Jens Elofsson <jens.elofsson@arm.com>

Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Signed-off-by: Jens Elofsson <jens.elofsson@arm.com>

Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Dan Johansson authored Mar 26, 2025 and Emil Ohlsson committed Mar 26, 2025

Optimizes this RHS packing by vectorizing the XOR operation. This is done
for segment lenghts of 4 or 8 bytes. The unoptimized path is used for
any other segment length.

Signed-off-by: Dan Johansson <dan.johansson@arm.com>

Approved-by: Emil Ohlsson <emil.ohlsson@arm.com>

* Alias the KleidiAI target for consistent linking to KleidiAI with `KleidiAI::kleidiai`.
* Add install instructions for the KleidiAI library target and its public headers.
* Export the KleidiAI target so that projects may use the imported target with `find_package()`.

Usage:

Install KleidiAI:
```
cmake -S . -B build
cmake --build build
cmake --install build
```

Once installed, KleidiAI can be imported and linked to using `find_package()`.

```
find_package(KleidiAI CONFIG REQUIRED)
target_link_libraries(my_framework PRIVATE KleidiAI::kleidiai)
```

Signed-off-by: Jakub Sujak <jakub.sujak@arm.com>

Approved-by: Viet-Hoa Do <viet-hoa.do@arm.com>

Jens Elofsson authored Mar 12, 2025 and Emil Ohlsson committed Mar 12, 2025

Update all version indicators to 1.5.0.

Signed-off-by: Jens Elofsson <jens.elofsson@arm.com>

Approved-by: Felix Johnny Thomasmathibalan <felixjohnny.thomasmathibalan@arm.com>

Anitha Raj authored Mar 12, 2025 and Emil Ohlsson committed Mar 12, 2025

Fix for reading LHS scale values from in kai_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa

Fix the out-of-bounds read while loading the scale values from LHS packed matrix in \`kai_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa\` by updating the predicate

Resolves: KLEIDIAI-507

Signed-off-by: Anitha Raj <anitha.raj@arm.com>

Reviewed-by: Viet-Hoa Do <viet-hoa.do@arm.com>
Reviewed-by: Emil Ohlsson <emil.ohlsson@arm.com>
Approved-by: Emil Ohlsson <emil.ohlsson@arm.com>

Signed-off-by: Jens Elofsson <jens.elofsson@arm.com>

Approved-by: Emil Ohlsson <emil.ohlsson@arm.com>

Jakub Sujak authored Feb 24, 2025 and Felix Johnny Thomasmathibalan committed Feb 24, 2025

* Refactor the benchmark tool to create a generic abstraction that
allows for running matrix multiplication micro-kernels with different
interfaces.

* Extend benchmark support to all matrix multiplication micro-kernels in
 the library.

Signed-off-by: Jakub Sujak <jakub.sujak@arm.com>

Approved-by: Felix Johnny Thomasmathibalan <felixjohnny.thomasmathibalan@arm.com>

Michael Kozlov authored Feb 20, 2025 and Felix Johnny Thomasmathibalan committed Feb 20, 2025

- Add new assembly ukernel optimized with FEAT_I8MM for matrix multiplication with 4x8 block size.
- Update build script.
- Add to unit test.

Signed-off-by: Michael Kozlov <michael.kozlov@arm.com>

Approved-by: Felix Johnny Thomasmathibalan <felixjohnny.thomasmathibalan@arm.com>

Jens Elofsson authored Feb 18, 2025 and Jakub Sujak committed Feb 18, 2025

Signed-off-by: Jens Elofsson <jens.elofsson@arm.com>

Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Update all version indicators to 1.4.0.

Signed-off-by: Jens Elofsson <jens.elofsson@arm.com>

Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Jens Elofsson authored Feb 18, 2025 and Felix Johnny Thomasmathibalan committed Feb 18, 2025

Signed-off-by: Jens Elofsson <jens.elofsson@arm.com>

Reviewed-by: Felix Johnny Thomasmathibalan <felixjohnny.thomasmathibalan@arm.com>
Approved-by: Felix Johnny Thomasmathibalan <felixjohnny.thomasmathibalan@arm.com>

Michael Kozlov authored Feb 17, 2025 and Felix Johnny Thomasmathibalan committed Feb 17, 2025

- Add new assembly ukernel optimized with FEAT_DOTPROD for matrix multiplication with 1x8 block size.
- Update build script.
- Add to unit test.

Signed-off-by: Michael Kozlov <michael.kozlov@arm.com>

Reviewed-by: Felix Johnny Thomasmathibalan <felixjohnny.thomasmathibalan@arm.com>
Approved-by: Felix Johnny Thomasmathibalan <felixjohnny.thomasmathibalan@arm.com>

Emil Ohlsson authored Feb 17, 2025 and Jakub Sujak committed Feb 17, 2025

Add support for GEMV like kernel for producing QAI8 from QAI8 LHS and
QSI8CXP packed RHS.

Update unit tests to include support for new kernel

Signed-off-by: Emil Ohlsson <emil.ohlsson@arm.com>

Reviewed-by: Felix Johnny Thomasmathibalan <felixjohnny.thomasmathibalan@arm.com>
Reviewed-by: Anton Bondarenko <anton.bondarenko@arm.com>
Reviewed-by: Emil Ohlsson <emil.ohlsson@arm.com>
Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Michael Kozlov authored Feb 12, 2025 and Emil Ohlsson committed Feb 12, 2025

- Update build script
- Add to unit test

Signed-off-by: Michael Kozlov <michael.kozlov@arm.com>

Reviewed-by: Anton Bondarenko <anton.bondarenko@arm.com>
Approved-by: Anton Bondarenko <anton.bondarenko@arm.com>

Anitha Raj authored Feb 11, 2025 and Anton Bondarenko committed Feb 11, 2025

Resolves: #KLEIDIAI-485, #COMPMID-8024

Signed-off-by: Anitha Raj <anitha.raj@arm.com>

Reviewed-by: Anitha Raj <anitha.raj@arm.com>
Approved-by: Anton Bondarenko <anton.bondarenko@arm.com>

Michael Kozlov authored Feb 06, 2025 and Jakub Sujak committed Feb 06, 2025

- Update build script
- Add to unit test

Signed-off-by: Michael Kozlov <michael.kozlov@arm.com>

Approved-by: Anitha Raj <anitha.raj@arm.com>

Suhail M authored Jan 31, 2025 and Felix Johnny Thomasmathibalan committed Jan 31, 2025

- Added demonstration of klediai integration as external dependency 

Signed-off-by: Mohammed Suhail Munshi <MohammedSuhail.Munshi@arm.com>

Reviewed-by: Mohammed Suhail Munshi <mohammedsuhail.munshi@arm.com>
Reviewed-by: Emil Ohlsson <emil.ohlsson@arm.com>
Approved-by: Emil Ohlsson <emil.ohlsson@arm.com>

Anton Bondarenko authored Jan 30, 2025 and Felix Johnny Thomasmathibalan committed Jan 30, 2025

Signed-off-by: Anton Bondarenko <anton.bondarenko@arm.com>

Approved-by: Felix Johnny Thomasmathibalan <felixjohnny.thomasmathibalan@arm.com>

Emil Ohlsson authored Jan 29, 2025 and Anton Bondarenko committed Jan 29, 2025

Update all version indications

Signed-off-by: Emil Ohlsson <emil.ohlsson@arm.com>

Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Michael Kozlov authored Jan 28, 2025 and Felix Johnny Thomasmathibalan committed Jan 28, 2025

Signed-off-by: Michael Kozlov <michael.kozlov@arm.com>

Signed-off-by: Anitha Raj <anitha.raj@arm.com>

Reviewed-by: Viet-Hoa Do <viet-hoa.do@arm.com>
Reviewed-by: Jakub Sujak <jakub.sujak@arm.com>
Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Anitha Raj authored Jan 27, 2025 and Jakub Sujak committed Jan 27, 2025

In kai_matmul_clamp_f32_qai8dxp1vlx8_qsi4cxp4vlx8_1vlx4vl_sme2_mopa:
 * Fix the offset calculation
 * Fix pointer increments in the matmul

Add new shapes to unit tests, to test n > 64

Resolves: #KLEIDIAI-405, #COMPMID-7918

Signed-off-by: Anitha Raj <anitha.raj@arm.com>

Reviewed-by: Viet-Hoa Do <viet-hoa.do@arm.com>
Reviewed-by: Anitha Raj <anitha.raj@arm.com>
Reviewed-by: Emil Ohlsson <emil.ohlsson@arm.com>
Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Signed-off-by: Emil Ohlsson <emil.ohlsson@arm.com>

Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Downstream projects might not build with same toolchain, which might
trigger warnings. Instead of treating these warnings as errors, using
this patch they are only warnings.

Also, add `-Werror` to the build flags to our testing pipeline, because
we do not want to accidentally introduce new warnings by mistake

Signed-off-by: Emil Ohlsson <emil.ohlsson@arm.com>

Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Mention recent fixes for compiler warnings produced by enabling the `-Wcast-qual -Wmissing-prototypes -Wstrict-prototypes -Woverlength-strings` compiler options.

Signed-off-by: Jakub Sujak <jakub.sujak@arm.com>

Approved-by: Emil Ohlsson <emil.ohlsson@arm.com>

Signed-off-by: Emil Ohlsson <emil.ohlsson@arm.com>

Approved-by: Felix Johnny Thomasmathibalan <felixjohnny.thomasmathibalan@arm.com>

Gian Marco Iodice authored Jan 16, 2025 and Emil Ohlsson committed Jan 16, 2025

- Previously, we were dividing m_idx/n_idx by m_step/n_step. However, since m_step/n_step can be different from mr/nr, we should divide by mr/nr instead

Signed-off-by: Gian Marco Iodice <gianmarco.iodice@arm.com>

Signed-off-by: Anitha Raj <anitha.raj@arm.com>

Reviewed-by: Anitha Raj <anitha.raj@arm.com>
Reviewed-by: Emil Ohlsson <emil.ohlsson@arm.com>
Approved-by: Emil Ohlsson <emil.ohlsson@arm.com>

Describe changes since 1.1.0

Signed-off-by: Emil Ohlsson <emil.ohlsson@arm.com>

Reviewed-by: Emil Ohlsson <emil.ohlsson@arm.com>
Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Signed-off-by: Jakub Sujak <jakub.sujak@arm.com>

Approved-by: Felix Johnny Thomasmathibalan <felixjohnny.thomasmathibalan@arm.com>

Michael Kozlov authored Dec 24, 2024 and Jakub Sujak committed Dec 24, 2024

- Add unit test

Signed-off-by: Michael Kozlov <michael.kozlov@arm.com>

Reviewed-by: Viet-Hoa Do <viet-hoa.do@arm.com>
Reviewed-by: Anitha Raj <anitha.raj@arm.com>
Reviewed-by: Felix Johnny Thomasmathibalan <felixjohnny.thomasmathibalan@arm.com>
Reviewed-by: Emil Ohlsson <emil.ohlsson@arm.com>
Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Anitha Raj authored Dec 24, 2024 and Jakub Sujak committed Dec 24, 2024

* GEMM and GEMV micro-kernels to compute the matrix multiplication of dynamically quantized 8-bit integer (QAI8DX) LHS matrix and quantized 4-bit integer (QSI4CX) RHS matrix and the accumulation of the result into a single-precision (F32) output, optimized for SME2 technology.

Signed-off-by: Mohamad Najem <mohamad.najem@arm.com>

Signed-off-by: Anitha Raj <anitha.raj@arm.com>

Signed-off-by: Michael Kozlov <michael.kozlov@arm.com>

Signed-off-by: Thomas Bamelis <thomas.bamelis@arm.com>

Reviewed-by: Anitha Raj <anitha.raj@arm.com>
Reviewed-by: Anton Bondarenko <anton.bondarenko@arm.com>
Reviewed-by: Jakub Sujak <jakub.sujak@arm.com>
Approved-by: Jakub Sujak <jakub.sujak@arm.com>

Changes reported release version, and updates changelog since last
release

Signed-off-by: Emil Ohlsson <emil.ohlsson@arm.com>

Reviewed-by: Emil Ohlsson <emil.ohlsson@arm.com>
Approved-by: Jakub Sujak <jakub.sujak@arm.com>