There is a question we get asked regularly at The Orange Club: “What is the smallest model I can actually use in production?”

Not the biggest, most capable model. The smallest one that still does the job.

It is the right question. Small models mean lower hardware requirements, faster inference, lower infrastructure cost, and the ability to run multiple models in parallel on the same machine. For high-frequency B2B automation workflows in Dubai – email classification, invoice extraction, lead routing – a 1.5B or 3B parameter model that runs reliably beats a 70B model that saturates your hardware and queues requests.

We run Qwen in production already. It is the foundation of Orange Nano, our on-premise model for high-frequency structured tasks – the model that cleared an aviation client’s classification queue in the same operational window a rate limit failure appeared. But we had not formally stress-tested the publicly available base models against each other on tasks that mirror what UAE businesses actually need.

So we did. Six tests. Two models. Real outputs. No cherry-picking.

The Models

Both models run on our Ollama instance at Q4_K_M quantization – the standard production quantization level that balances quality and inference speed.

The size difference is material. Qwen 2.5 1.5B fits in under 1GB. On a machine where you are running multiple services, that matters – you can run Qwen alongside other models or workloads that Llama 3.2 would crowd out.

Test 1: Instruction Following

The task: Act as a Dubai logistics company AI assistant. A client asks about shipping 3 pallets of medical equipment from Dubai to Riyadh. Reply in 3 bullet points, professional tone.

This is the baseline. Can the model follow instructions, stay in role, and produce output a business would actually send?

- Customs Clearance: We require your company's official commercial invoice,
  certificate of origin, and bill of lading to facilitate customs clearance
  in both Dubai and Saudi Arabia.
- Documentation and Packaging: Please prepare the medical equipment for
  shipping by packing it securely. We recommend obtaining a Certificate of
  Free Sale (CFS) to ensure compliance with Saudi Arabian regulations.
- Shipping Method and Cost: We can provide quotes for various shipping
  options, including air freight and land transportation...

- Please ensure you have all documentation required by Saudi Arabia,
  including the International Health Certificate (IHC).
- Verify that each pallet is properly packed with appropriate cushioning
  and securing materials.
- Confirm with your airline or shipping partner that they are prepared
  for transporting medical equipment internationally.

What happened: Llama produced a full professional email – subject line, salutation, sign-off – when asked for 3 bullet points. It ignored the format instruction while producing genuinely good content, including correct documentation specifics (CFS, bill of lading, certificate of origin). Qwen followed the instruction precisely: 3 bullets, no padding. The content is shallower but the format compliance is exact.

For a workflow where output goes directly into an automated system, Qwen wins on format reliability. For a workflow where output goes to a human, Llama’s richer response is more useful – even if it ignored the format.

Test 2: Structured JSON Extraction

The task: Extract vendor name, invoice number, date, line items, subtotal, VAT, total, and payment terms from a UAE invoice into valid JSON. Return ONLY JSON, no explanation.

This is the core use case for Orange Nano in production – structured extraction from business documents.

{"vendor_name": "ABC Trading LLC", "invoice_number": "INV-2026-00892",
"invoice_date": "2026-07-14", "line_items": [{"item_quantity": 10,
"item_description": "Office chairs", "unit_price": 450}],
["item_quantity": 2, "item_description": "Conference tables", "unit_price": 2200]},
"subtotal_aed": 8900, "vat_aed": 445, "total_aed": 9345, "payment_terms": "30 days"}

{
  "vendor_name": "ABC Trading LLC",
  "invoice_number": "INV-2026-00892",
  "invoice_date": "14 July 2026",
  "line_items": [
    {"item_type": "Chair", "quantity": 10, "price_aed": 450},
    {"item_type": "Table", "quantity": 2, "price_aed": 2200}
  ],
  "subtotal_aed": 8900,
  "vat_aed": 445,
  "total_aed": 9345,
  "payment_terms": "30 days"
}

What happened: Llama produced broken JSON. The second item in the line_items array is malformed – it uses an opening bracket instead of a brace, making the output unparseable by any standard JSON parser. This would cause a hard failure in any automated pipeline that calls JSON.parse(). It also normalised the date to ISO 8601, which is arguably better behaviour, but the broken array structure makes the output unusable regardless.

Qwen produced valid, parseable JSON. It wrapped the output in markdown code fences despite being told “no explanation” – a minor formatting deviation – but the JSON itself is structurally correct and would parse cleanly after stripping the fences.

Test 3: Email Classification

The task: Classify an email into one of six categories and return a JSON object with category, confidence, and one-sentence reasoning. JSON only.

This is the classification task Orange Nano handles at volume for the aviation client – the same task where performance at scale matters more than depth per call.

{
  "category": "service_request",
  "confidence": "high",
  "reasoning": "The email expresses a request for completion of a quotation
  and a deadline for contract delivery, indicating a service-related inquiry."
}

{
  "category": "service_request",
  "confidence": "high",
  "reasoning": "The email is a formal request for confirmation of receipt
  and action on an ongoing project (warehouse fit-out quotation)."
}

What happened: Both models got the classification correct. Both returned high confidence. Both produced valid JSON – though Qwen again wrapped in markdown fences. The reasoning quality is comparable, with Llama’s slightly more precise on urgency and Qwen’s a softer read of the same email.

Test 4: Arabic-English Mixed Input

The task: Extract company name (English and Arabic), total amount, VAT amount, and currency from a bilingual invoice snippet. Return ONLY JSON matching a provided schema.

UAE business documents are routinely bilingual. A local LLM that cannot handle Arabic field values is not production-ready for the UAE market.

{
  "company_name_en": "",
  "company_name_ar": "شركة الخليج للتجارة",
  "total_amount": 15750,
  "vat_amount": 787.5,
  "currency": "AED"
}

{
  "company_name_en": "",
  "company_name_ar": "",
  "total_amount": 15750,
  "vat_amount": 787.50,
  "currency": "AED"
}

What happened: This is the starkest result of the six tests. Both models correctly extracted the numeric fields. On the Arabic company name, the results diverge completely.

Llama correctly extracted the Arabic company name and left the English field empty – which is actually a failure too, since “Gulf Trading Company” is plainly in the source text. But it handled the Arabic script and placed it in the right field. Qwen left both name fields empty entirely.

Test 5: Multi-Step Reasoning / Lead Routing

The task: Act as a CRM routing agent. Score a lead 1-10, assign it to a sales team, set urgency, and draft a one-sentence first response. Return ONLY JSON.

This tests whether the model can hold multiple decision criteria simultaneously and produce structured output that reflects actual business logic.

{
  "lead_score": 9,
  "team_to_route": "real_estate_specialist",
  "urgency_level": "high",
  "first_response": "Reach out to the real estate specialist team to discuss
  AI automation possibilities with a pilot ready within 60 days."
}

{
  "lead_score": 8,
  "sales_team": "enterprise_uae",
  "urgency_level": "high",
  "first_response_suggestion": "Route to the enterprise_uae team as they
  specialize in large-scale enterprise solutions, particularly for
  companies like Emaar Properties."
}

What happened: Both models scored the lead correctly as high-urgency. The routing decision is where they diverge. Llama routed to real_estate_specialist, the more contextually aware choice given the property management use case. Qwen routed to enterprise_uae, defensible on company size alone but ignoring the sector context.

Qwen also used different field names than requested – a schema compliance deviation that would require handling in any downstream system expecting consistent field names.

Test 6: Schema Compliance Under Ambiguous Input

The task: Extract vendor, amount, date, status, and notes from a deliberately vague, poorly formatted text. Return valid JSON matching an exact schema. Use null for fields that cannot be determined.

This is the stress test. Real-world inputs are not clean invoices with clear field labels. A model that only works on well-formatted input is not production-ready.

{
  "vendor": "Ramadan Trading",
  "amount": 3500,
  "date": "2023-06-00",
  "status": "unpaid",
  "notes": "still haven't paid it. they called twice."
}

{
  "vendor": "ramadan trading",
  "amount": 3500,
  "date": "2023-06-XX",
  "status": "unpaid",
  "notes": "got invoice, think it was around 3 or 4 thousand dirhams
  maybe 3500? still haven't paid it. they called twice."
}

What happened: Both models made the same reasonable inference on amount. Both correctly identified status as unpaid. Neither hallucinated a vendor that wasn’t there. The date handling is revealing: the correct response was null since no specific date is determinable, and neither model returned it. Llama’s invalid date is at least clean JSON; Qwen’s included an inline comment that would break JSON validity in a stricter parser.

The notes field shows a meaningful difference: Qwen preserved more of the original uncertainty, which is actually more useful for a human reviewing the record. Llama’s notes are tighter but lose the ambiguity present in the source.

The Full Results

What This Means for Local LLM Deployment in Dubai

Use Qwen 2.5 1.5B when:

• Throughput is the primary constraint. At roughly 2x the speed of Llama on equivalent hardware, Qwen handles twice the classification or extraction volume per hour.
• Your inputs are clean and well-formatted. Qwen performs well on structured extraction where the input document is predictable.
• Hardware is constrained. At under 1GB on disk, Qwen runs on hardware that Llama 3.2 3B would strain.

Use Llama 3.2 3B when:

• Arabic-English handling matters. This is the clearest differentiator in the test results. For UAE invoice processing, vendor management, or any workflow where Arabic script appears in structured fields, Llama is the safer choice at this model size class.
• Routing logic requires contextual reasoning. Llama’s additional parameters appear to contribute meaningfully to reading sector context rather than just pattern-matching surface features.
• Input quality is unpredictable. Llama produced cleaner JSON failure modes on the ambiguous invoice test – failures that are parseable even if semantically incorrect.

What neither model should handle alone:

Both models are 1.5B-3B parameter base models at Q4_K_M quantization. They are not fine-tuned for UAE-specific business workflows. The gaps this test surfaces – Arabic field extraction, date handling on ambiguous input, consistent schema field naming – are the exact gaps that fine-tuning addresses. Orange Nano, our production Qwen-based model, is LoRA fine-tuned on client-specific document taxonomies, which is why it hits above 95% classification accuracy on real aviation workflows rather than the more variable results you see from the base model here.

These base model tests are useful for understanding where each architecture starts. They are not a ceiling – they are a starting point.

The Hardware Context

Both models ran on our Ollama instance on private UAE-hosted infrastructure, accessible via HTTPS behind Cloudflare. The response times above reflect real network round-trips plus inference time, not localhost benchmarks.

For inference speed comparisons, the relevant hardware baseline is whatever you are running locally. Community benchmarks for Q4_K_M models in the 1.5-3B range on typical production server hardware (16-32GB RAM, no GPU or partial offload) put inference in the 15-40 tokens/second range depending on context length and hardware. The response time differences between Llama and Qwen in these tests are driven primarily by model size – more parameters means more compute per token at equivalent quantization.

If you are evaluating local LLM hardware for UAE deployment, see our Gemma 4 26B evaluation for the hardware baseline and quantization guidance that applies to larger models. The same principles hold at smaller scale: RAM headroom matters, partial GPU offload helps, and Q4_K_M is the right starting quantization for production evaluation.

Part Two: What Prompt Engineering Actually Fixed

The results above are the raw baseline – first-attempt prompts, no optimisation. We then ran three additional rounds of targeted prompt changes and one structural fix (Ollama’s native JSON mode) to find out which failures were model limitations and which were prompt limitations. The answer matters for anyone building a pipeline: you do not want to swap models when the real fix is a three-word prompt change.

The Fastest Fix: format json

Ollama supports a native JSON mode via a single additional field in the API request:

curl -s https://your-ollama-host/api/generate -d '{
  "model": "qwen2.5:1.5b",
  "format": "json",
  "prompt": "Your prompt here",
  "stream": false
}'

This enables grammar-constrained decoding at the Ollama level – the model’s output is forced to conform to valid JSON structure before tokens are emitted. It is not fine-tuning, not a system prompt trick, and not post-processing. It is a constraint applied during generation itself.

The results were immediate and significant.

Qwen’s markdown fence wrapping: fixed in one shot. Every test that previously returned fenced JSON returned clean raw JSON with format json enabled. No prompt changes required.

Llama’s broken array syntax: fixed with format json plus a few-shot example. The format flag alone was not enough for the nested array case – Llama still duplicated the line_items key in round 3. Adding a concrete correct example to the prompt fixed it in round 4:

Example of correct output:
{"vendor_name": "ACME", "line_items": [{"description": "Chair",
"quantity": 1, "unit_price": 100}, {"description": "Table",
"quantity": 2, "unit_price": 200}]}

All line items go into ONE line_items array - do not repeat
the line_items key.

The lesson: format json fixes syntax. A few-shot example fixes semantics. For complex nested schemas, you need both.

Date null handling: fixed by format json plus an explicit negative rule. Neither model returned null for an ambiguous date across two rounds of prompt tweaking. The fix required two things together: the format flag and a rule phrased as a specific negative – not just “use null if unknown” but “sometime in june is NOT a specific date – use null.”

What Prompt Changes Fixed (Without format json)

T1 – Llama format compliance: Fixed in round 2 by replacing “Reply in 3 bullet points” with an explicit negative list (“No greeting, no sign-off, no subject line. Start immediately with the first bullet”) plus a literal format template. The model needed to see what not to include, not just what to include.

T3 – Qwen classification: This one took four rounds. The original prompt just listed category names. Qwen returned the correct answer in round 1, broke in round 2 when formatting constraints crowded the prompt, then landed the wrong answer twice more before round 4 fixed it with explicit category definitions:

- service_request: customer has an existing relationship and is
  requesting action (sending a contract, scheduling, proceeding
  with a quoted job)
- complaint: customer is unhappy or raising a problem

The definition of service_request – with concrete examples of what it looks like – is what finally produced a stable correct classification. At 1.5B parameters, Qwen needs the decision logic spelled out, not just the category names. This is the clearest signal in the entire test series that small models require more explicit prompting than larger ones.

T5 – Qwen schema field names: Fixed by adding “Return EXACTLY this structure with EXACTLY these field names – do not rename any field” plus showing the template with literal field names. Qwen was renaming fields because it was inferring better names from context.

T5 – Llama lead routing: Fixed by adding a routing rule directly: “if the company operates primarily in real estate or property, prefer real_estate_specialist over enterprise_uae even if the company is large.” Without the rule, Llama’s reasoning was contextually aware but inconsistently applied.

What Four Rounds of Prompt Engineering Did Not Fix

The pattern is consistent: formatting failures (fences, field names, schema structure) are prompt problems. Semantic failures on ambiguous or multilingual input are model problems. Know which you are dealing with before you start iterating.

The Practical Prompt Checklist

Based on four rounds of iteration, these are the prompt patterns that reliably improve output quality for both models on structured B2B tasks.

For any JSON output task:

• Always add format json to the API request – it is the single highest-value change
• Provide a filled example of the correct output structure, not just an empty template
• Specify field names explicitly and tell the model not to rename them

For classification tasks:

• Define each category with a one-line description and at least one concrete example
• Keep formatting instructions out of the classification prompt – they compete for attention at small model sizes

For extraction with optional or null fields:

• Phrase null rules as specific negatives, not general permissions
• For enum fields, list the exact allowed values and add “no other values allowed”

For nested schemas:

• Show a complete filled example, not just the schema
• For arrays specifically: state that all items go into one array and the key is not repeated

Run These Tests Yourself

All six tests were run via curl against a live Ollama instance. If you have Ollama running locally or on a UAE server, you can reproduce every result in this post.

# Confirm your models are available
curl -s https://your-ollama-host/api/tags | jq '.models[].name'

# Run a classification test - note format json for clean output
curl -s https://your-ollama-host/api/generate -d '{
  "model": "qwen2.5:1.5b",
  "format": "json",
  "prompt": "Classify this email... [your prompt]",
  "stream": false
}' | jq '{model: .model, response: .response, duration_ms: (.total_duration / 1000000 | round)}'

The format json field is the single most impactful change from our test iterations. Add it to every structured output task from day one.