Should I preload my models on download or download after my container is started?

The optimal approach depends on the specific application you are running. However, there are some guidelines and considerations to help you decide whether to download your application models on the node after the container is running or to include them in your container image.

Why would I include the model(s) in my container image?

1. Your container image could be more efficient to manage.
2. Your application would not be dependent on external model servers and the network connection to those servers.
3. The model is loaded onto the node while pulling the image, which is not billed. So included your model in the container image could be more cost effective.

Why would I download the model(s) after the instance starts?

1. You have the opportunity to do suitability checks on the node before spending time downloading the model.
2. You can load models dynamically, without knowing the model at the time the container is deployed.
3. There is no need to manage multiple containers or a large container for multiple models.

Why would I use a hybrid approach?

You can include popular models in your container image, and then dynamically download other models as needed once the container starts.

1. Container images are capped at 35GB on the Salad network. Users that support large numbers of models might find it impractical or impossible to include them all with Salad's container size limitations.
2. Preloading with your most popular models allows the container to become productive right after it reaches a running state, while downloading additional models in the background as needed.