With all this talk about large language models, you’d be forgiven for thinking that they’re going to solve the world’s problems. All we need is more data and more computing power!
However, when it comes to enterprise AI, bigger isn’t always better. It’s true that large language models have some brilliant capabilities, but do you *really* need a large language model for your use cases? Perhaps a small language model will do.
What is a small language model (SLM)?
A small language model is an AI model, similar to a large language model, only with less training data and less parameters. They fundamentally do the same thing as a large language model; understand and generate language, but are smaller and less complex.
How big is a small language model?
Small language models come in a variety of shaped and sizes and the definition of when a model becomes a large language model differ depending on who you ask. Typically, though, anything below 30 billion parameters is considered a small language model. However, SMLs can be as small as a few hundred million parameters.
What’s the difference between small language model (SLM) and a large language model (LLM)?
There are 10 primary differences between the two that will help you understand which type of model you might consider for a given use case:
- Size. This is obvious. As mentioned above, LLMs are a lot larger than SLMs. Some of the more recent LLMs such as Claude 3 and Olympus, have 2 trillion parameters! Compare that with Phi-2 at 2.7 billion.
- Training data. LLMs require extensive, varied data sets for broad learning requirements. SLMs use more specialist and focused, smaller data sets.
- Training time. To train an LLM, it can take months. SLMs can be trained in weeks.
- Computing power and resources. Because of the large data sets and parameter sizes, LLMs consume a LOT of computing resource to train and run the models. SLMs use far less (still a lot, but less), making them a more sustainable option.
- Proficiency. LLMs are typically more proficient at handling complex, sophisticated and general tasks. SLMs are best for more adequate, simpler tasks.
- Adaptation. LLMs are harder to adapt to customised tasks and require heavy lifting for things like fine tuning. SLMs are much easier to fine tune and customise for specific needs.
- Inference. LLMs require specialised hardware, like GPUs, and cloud services to conduct inference. This means they have to be used over the internet. SLMs are so small, they can be ran locally on a raspberry pi or a phone, meaning they can run without an internet connection.
- Latency. If anyone’s tried building a voice assistant with an LLM, then you’ll know that latency is a huge issue. Depending on the task, you’re waiting seconds for LLMs to respond. SLMs, because of their size, are typically much quicker.
- Cost. Inevitably, if you’re having to consume a lot of computing resource for inference, and your model size is bigger, it means that the token cost for LLMs is high. For SLMs, it’s a lot lower, meaning they’re cheaper to run.
- Control. With LLMs, you’re in the hands of the model builders. If the model changes, you’ll have drift or worse, catastrophic forgetting. With SLMs, anyone can literally run them on your own servers, tune them, then freeze them in time, so that they never change.
How to decide what sized model to use
To decide what mode to to use, first start experimenting with large language models. This is to validate that the task you’re trying to accomplish can, in fact, be done. If it can be done at all, an LLM should be able to do it.
Once you’ve proven that the task is doable, you can then start working down in model sizes to figure out whether the same task can be done using a smaller model. When you reach a model size where your results start to change, get less accurate or slightly more unpredictable, you’ve reached your potential model size.
That doesn’t necessarily mean you should go back up in model size. It may mean that the model size you’ve reached requires some further tuning or training.
Prompt tuning
This tuning can be done firstly with prompt tuning. This is specifically to provide the model with some in-context learning i.e. data that it can use to accomplish the task, delivered to it in the prompt.
Retrieval augmented generation
Second, consider retrieval augmented generation (RAG) or *indexing*. This is to provide the model with external data that is can use at runtime to pull into its responses. For some use cases, especially those that are search-based of some kind, you’ll find this may give you the results you’re looking for and is easier than fine tuning as it doesn’t require tampering with model weights or access to the raw model.
Fine tuning
Lastly, if the first two options haven’t solved your problem, then fine tuning is the final consideration. This is where you train the model for a specific task based on data related to that specific task. There are a number of different types of fine tuning methods you can use, ranging from fine tuning the output using embeddings all the way through to fine tuning the parameters of the models themselves. To do this, you need access to the raw model, rather than an API, and so you’re typically heading into small language model territory here. Find out more about fine tuning.
Relevance for enterprise teams
Think about the tasks that you have for AI within your organisation. It’s probably something along the lines of:
- Intent classification
- Knowledge retrieval
- Content summarisation
- Sentiment analysis
- Conversation management
- Contextual response generation
- Translation
And things like that. Of course, there are many more use cases for AI, but these are among the most common.
Now consider whether you need the intense power of a large language model for these tasks.
Classification? Really? You need all the internet’s information to be able to recognise that ‘my credit card was stolen’ means ‘stolen card’?
The *vast* majority of enterprise AI requirements are specific to that enterprise. You more than likely don’t need the most powerful AI tools on the planet to do what you want (and you certainly don’t need the cost).
Try working backwards from a large language model and see whether a small language model is more fit for your purpose.
