Earlier today, I was helping my daughter with her math homework which involved prime numbers. She questioned if 23 was prime. I attempted to prove to her a few times that it was, but to verify she quickly asked "Alexa - is 23 a prime number". The response was immediate and short, "Yes, 23 is a prime number."
Given Alexa's fast and correct response, I thought it'd be an interesting test to see what various Large Language Models (LLM) would generate. To my surpise, most of the models I tried got this simple prompt wrong 9 out of 10 times. It is generally understood that LLMs are not good a math, but I assumed a simple question as this would be straight forward.
To test a few things, I made a quick Jupyter Notebook to loop over a few models and output their responses.
The first test was simply "is 23 a prime number?", and I also tried asking a model to first list prime numbers and then if 23 is a prime number seems to greatly affect the accuracy of the response.
llama2:13b-chat
What are the first 15 prime numbers? Is 23 a prime number?
Here are the first 15 prime numbers: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43
No, 23 is not a prime number. It can be divided by 3 and 7, so it is a composite number.
Is 23 a prime number?
Yes, 23 is a prime number.
mistral:7b-instruct
What are the first 15 prime numbers? Is 23 a prime number?
The first 15 prime numbers are: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43
Yes, 23 is a prime number. It only has two distinct positive divisors: 1 and itself.
Is 23 a prime number?
No, the number 23 is not a prime number in its simplest form. A prime number is a natural number greater than 1 that has no positive divisors other than 1 and itself. The number 23 does have one positive divisor other than 1, which is 23 itself, but it also has another divisor, namely 1. So, although 23 is a prime number in the sense that it cannot be divided evenly by any number smaller than itself, it fails to meet the more rigorous definition of a prime number as having no divisors other than 1.
codellama:13b
What are the first 15 prime numbers? Is 23 a prime number?
The first 15 prime numbers are:
- 2
- 3
- 5
- 7
- 11
- 13
- 17
- 19
- 23
- 29
- 31
- 37
- 41
- 43
- 47
Yes, 23 is a prime number.
Is 23 a prime number?
No, 23 is not a prime number. It can be divided by 1, 3, and 7, so it has factors other than 1 and itself.
gemma:7b-instruct
What are the first 15 prime numbers? Is 23 a prime number?
Sure, here are the first 15 prime numbers:
2, 3, 5, 7, 11, 13, 17, 19, 21, 23, 25, 29, 31, 33, 35, 37
23 is a prime number.
Is 23 a prime number?
No, 23 is not a prime number. Prime numbers are numbers that are divisible by only 1 and themselves. 23 is divisible by 1, 23, and 46, so it is not prime.
Interestly most models got the simple prompt incorrect, but did get the correct answer when asked to generate a list of primes first. Llama was the only model that some how managed to provide the correct reponse to the simple prompt, but disagree with itself on the list prompt.
Next I tried a prompt asking if 23 is in a list of prime numbers, rather then directly asking if it were a prime. I also added a completion technique of getting it to complete a statement instead of asking it a direct question.
llama2:13b-chat
Is 23 in a list of prime numbers?
No, 23 is not a prime number. It can be divided by 1 and 23, so it is a composite number.
The statement "23 is a prime number" is
The statement "23 is a prime number" is false.
23 is a composite number, not a prime number. It can be divided by 1 and 23, so it is not a prime number.
gemma:7b-instruct
Is 23 in a list of prime numbers?
No, 23 is not in a list of prime numbers. Prime numbers are 2, 3, 5, 7, 11, 13, 17, 19, and 21.
The statement "23 is a prime number" is
The statement "23 is a prime number" is false. 23 is not a prime number. Prime numbers are numbers that are divisible by only 1 and themselves.
codellama:13b
Is 23 in a list of prime numbers?
No, 23 is not a prime number. It can be divided by 1, 3, and 23, so it is a composite number.
The statement "23 is a prime number" is
False. 23 is not a prime number because it can be divided by 1, 3, and 23, which means that it has more than one factor.
mistral:7b-instruct
Is 23 in a list of prime numbers?
No, the number 23 is not a prime number. A prime number is a natural number greater than 1 that has no positive divisors other than 1 and itself. While 23 is an odd prime, it can be written as the product of two prime numbers: 23 = 1 × 23, so it does not meet the definition of a prime number. The first few prime numbers are 2, 3, 5, 7, 11, and 13.
The statement "23 is a prime number" is
The statement "23 is a prime number" is true. A prime number is a natural number greater than 1 that has no positive divisors other than 1 and itself. The number 23 has only two positive divisors: 1 and 23, so it is a prime number.
llama2:13b-chat
Is 23 in a list of prime numbers?
No, 23 is not a prime number. It can be divided by 1 and 23, so it is a composite number.
The statement "23 is a prime number" is
The statement "23 is a prime number" is false.
23 is a composite number, not a prime number. It can be divided by 1 and 23, so it is not a prime number.
gemma:7b-instruct
Is 23 in a list of prime numbers?
No, 23 is not in a list of prime numbers. Prime numbers are 2, 3, 5, 7, 11, 13, 17, 19, and 21.
The statement "23 is a prime number" is
The statement "23 is a prime number" is false. 23 is not a prime number. Prime numbers are numbers that are divisible by only 1 and themselves.
codellama:13b
Is 23 in a list of prime numbers?
No, 23 is not a prime number. It can be divided by 1, 3, and 23, so it is a composite number.
The statement "23 is a prime number" is
False. 23 is not a prime number because it can be divided by 1, 3, and 23, which means that it has more than one factor.
mistral:7b-instruct
Is 23 in a list of prime numbers?
No, the number 23 is not a prime number. A prime number is a natural number greater than 1 that has no positive divisors other than 1 and itself. While 23 is an odd prime, it can be written as the product of two prime numbers: 23 = 1 × 23, so it does not meet the definition of a prime number. The first few prime numbers are 2, 3, 5, 7, 11, and 13.
The statement "23 is a prime number" is
The statement "23 is a prime number" is true. A prime number is a natural number greater than 1 that has no positive divisors other than 1 and itself. The number 23 has only two positive divisors: 1 and 23, so it is a prime number.
Only Mistral provided a correct response to the completion prompt. All models failed with the list question.
Lastly I decided to build a more indepth prompt, defining a goal and asking the models to use steps. This method shows the best results, however this shows the complexity of using LLMs compared to optimized assistants like Alexa.
llama2:13b-chat
You are a math expert that determines if a number is prime or not. You base your decision on your knowledge of existing prime numbers. Think about the answer step by step. Reply to the following question: "Is 23 a prime number?"
As a math expert, I can determine whether a number is prime or not based on my knowledge of existing prime numbers. To decide if 23 is a prime number, I will follow these steps:
Step 1: Check if 23 is divisible by any known prime numbers.
I know that the prime numbers up to 22 are 2, 3, 5, 7, 11, and 13. Therefore, I can check if 23 is divisible by any of these prime numbers.
Step 2: Check if 23 is divisible by 2.
2 does not divide 23, so 2 is not a factor of 23.
Step 3: Check if 23 is divisible by 3.
3 does not divide 23, so 3 is not a factor of 23.
Step 4: Check if 23 is divisible by 5.
5 does not divide 23, so 5 is not a factor of 23.
Step 5: Check if 23 is divisible by 7.
7 does not divide 23, so 7 is not a factor of 23.
Step 6: Check if 23 is divisible by 11.
11 does not divide 23, so 11 is not a factor of 23.
Step 7: Check if 23 is divisible by 13.
13 does not divide 23, so 13 is not a factor of 23.
Since none of the known prime numbers up to 22 divides 23, I can conclude that 23 is a prime number. Therefore, my answer is:
Yes, 23 is a prime number.
gemma:7b-instruct
You are a math expert that determines if a number is prime or not. You base your decision on your knowledge of existing prime numbers. Think about the answer step by step. Reply to the following question: "Is 23 a prime number?"
Step 1: Identify the multiples of 23.
To determine if 23 is prime, I first find its multiples. Multiples of 23 are 23, 46, 69, 92, and so on.
Step 2: Check if 23 is divisible by any prime number other than itself.
If 23 is divisible by any prime number other than itself, it is not prime. However, 23 is not divisible by any prime number other than itself.
Therefore, 23 is a prime number.
codellama:13b
You are a math expert that determines if a number is prime or not. You base your decision on your knowledge of existing prime numbers. Think about the answer step by step. Reply to the following question: "Is 23 a prime number?"
As a math expert, I can determine whether 23 is a prime number based on my knowledge of existing prime numbers.
To determine if 23 is a prime number, we need to check if it can be divided by any other number except for 1 and itself. Since 23 is not divisible by any other number except for 1 and itself, it is a prime number. Therefore, the answer is yes, 23 is a prime number.
mistral:7b-instruct
You are a math expert that determines if a number is prime or not. You base your decision on your knowledge of existing prime numbers. Think about the answer step by step. Reply to the following question: "Is 23 a prime number?"
Yes, 23 is a prime number. A prime number is a natural number greater than 1 that has no positive divisors other than 1 and itself. The first few prime numbers are 2, 3, 5, 7, 11, and 13. We can see that 23 only has two positive divisors: 1 and 23 itself, so it is a prime number.