Open Internet by MindsNet

Optimizing Transformer Model Size and Inference Speed Beyond Current Limits

The author is seeking guidance on optimizing a transformer-based neural network for inference speed and model size. Despite applying techniques like FP16 conversion, ONNX Runtime optimization, and pruning, the model size remains around 162 MB. The author needs advice on the next steps to achieve significant reductions in model size and improvements in inference speed.

Computing & Technology, Computer Science, Artificial Intelligence