MLBEDSW-7458: Fused activation not passed through correctly

- Fixed a problem where the fused activation got lost
when the op was passed through to the CPU
- The fix is to always make sure the attribute is not removed

Change-Id: I612cfa8f6f0a0465459080762094fe61e7ddc1c3
Signed-off-by: Johan Alfven <johan.alfven@arm.com>