Skip to content
Commit 1c590489 authored by Tim Hall's avatar Tim Hall
Browse files

MLBEDSW-7281: create_const_tensor OverflowError on Microsoft Windows



 - Additional overflow checks are performed when running under
Microsoft Windows compared to Linux. These checks happen when
converting from Python int to NumPy int/uint
 - The problem is that the lut activation values are int32 type,
however they are defined as Python ints. If these are converted to
numpy.int32 it could result in an overflow error
 - The fix is to convert these values to uint32 but keep the
operator's IFM tensor type the same (as this will allow them to be
interpreted correctly)
 - Fixing this highlighted another problem where convert_to_lut
always calls create_lut_tensor() with an int8 datatype, whereas it
should be using the IFM datatype

Change-Id: I781a9d850f654267aa4a67754438607c4bb95685
Signed-off-by: Tim Hall's avatarTim Hall <tim.hall@arm.com>
parent 090f18a5
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment