Skip to content
Five.Reviews
Menu

AI Tools & Comparisons

NVIDIA Nemotron 3.5 Bolt: Local Setup and Benchmarks

Analytics dashboard on a laptop screen used to represent evidence-led software evaluation
Free browser-based audio. No tracking or paid API required.

Running a capable AI model on your own computer should not take an afternoon of mismatched downloads and failed launches. NVIDIA Nemotron 3.5 Bolt is a useful test case because it comes in two distinct models: a 74B-A9B model designed for agent work within a 24GB memory target, and a smaller E4B model built for fast conversational agents. This guide explains how to choose the right variant, check your hardware, run the 74B-A9B model with llama.cpp, and benchmark it on your own system.

Quick Summary

Nemotron 3.5 Bolt is NVIDIA’s open-weight local model family, released in October 2026. The 74B-A9B 24GB model has 74 billion total parameters, 9 billion active per token, and uses 2-bit quantization to target a 24GB memory budget. The E4B 4GB model is an 8-billion-parameter text model built on Gemma 4 E4B, aimed at voice assistants and tool-using chat agents.

For agent and coding work on a 24GB-class NVIDIA GPU, start with the 74B-A9B model. For limited video memory and conversational use, start with E4B. In both cases, confirm the exact checkpoint’s requirements and supported runtime before downloading, because memory targets are not guarantees.

What Is NVIDIA Nemotron 3.5 Bolt?

NVIDIA describes Nemotron as a family of open models with open weights, training data, and recipes. The 74B-A9B model was compressed from Nemotron 3.5 Super VL, a 120-billion-parameter model, and reduces the number of experts used per token from 22 to 9. It accepts text, images, and video as input and generates text.

A mixture-of-experts (MoE) model contains many specialist sub-networks, called experts, and routes each token through only a few of them. The “9B active” figure describes compute per token. All 74 billion parameters must still be stored, so active parameter count does not equal memory use. Memory depends on total weights, quantization (storing weights in fewer bits), context length, and the key-value (KV) cache, which grows as conversations get longer.

E4B is a separate model with 8 billion total parameters, about 4 billion effective, post-trained from Google’s Gemma 4 E4B. It accepts text input and produces text output, and NVIDIA positions it for voice assistants, interactive characters, and tool use.

Which Model Should You Choose?

Factor74B-A9B 24GBE4B 4GB
ArchitectureVision-language MoE, 74B total, 9B activeGemma 4 E4B post-trained, 8B total, 4B effective
InputsText, images, videoText
ContextUp to 256K tokensUp to 64K tokens
Memory targetBelow 24GB at batch size 1 and 128K contextAbout 4GB in target workloads
llama.cpp pathNVIDIA GGUF repositoryQuick Start (page marks recipe as pending)
Main considerationLarge download; image input adds memoryTarget is not total system memory

A memory target describes what a model was designed to fit under stated conditions. Your actual minimum depends on context length, runtime overhead, and whether other programs share the GPU.

Choose the 74B-A9B model if you have a 24GB-class NVIDIA GPU, enough system RAM, and agent or coding tasks. Choose E4B for lightweight assistants on smaller GPUs. Check the 74B technical page and E4B technical page for the latest specifications.

Prerequisites

You need an NVIDIA GPU with a current driver, plus Git, CMake, and a C++ compiler to build llama.cpp. Check your GPU with:

bash

nvidia-smi

The output shows the GPU name, driver version, CUDA version, and total and used memory. It confirms the hardware is present but does not prove the model will load or run well.

The 74B model card is inconsistent about supported hardware. Its deployment summary lists vLLM on Ampere (A100), Hopper (H100-80GB), or Blackwell, while its software section lists only Blackwell SM120 and SM121. Verify your GPU against the current card before buying or renting hardware.

NVIDIA’s pages do not state a system RAM requirement for the 74B model. A community guide estimates 24 to 32GB of system RAM for short-context use, which NVIDIA has not confirmed. Check file sizes on each repository’s Files tab before downloading, since the model pages do not list them.

How to Run NVIDIA Nemotron 3.5 Bolt Locally

This tutorial covers the 74B-A9B 24GB model with llama.cpp, using NVIDIA’s official GGUF repository. A note at the end covers E4B.

Step 1: Download the model files

The repository provides the main language model and a vision projector (mmproj) that enables image input.

bash

pip install -U huggingface_hub

hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF \

  NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \

  mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \

  –local-dir models

NVIDIA also publishes a LiLiCorr drafter for speculative decoding in a separate repository. This guide omits it because the sources I reviewed do not document a verified llama.cpp configuration for it.

Step 2: Install llama.cpp

On Windows, run winget install llama.cpp. On macOS, run brew install llama.cpp. On Linux, build from source:

bash

git clone https://github.com/ggml-org/llama.cpp

cd llama.cpp

cmake -B build

cmake –build build -j –target llama-server llama-cli

To run the model on your GPU, add the CUDA option described in the llama.cpp repository to the cmake -B build step. One community GGUF page states that this model requires llama.cpp at commit cd0a5b6a1 or later. If the model fails to load, update your build first.

Step 3: Launch the server

bash

./build/bin/llama-server \

  -m models/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \

  –mmproj models/mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \

  -c 32768

The -m flag loads the language model, –mmproj loads the vision projector, and -c sets the context window in tokens. Start at 32768. NVIDIA’s memory target assumes up to 128K context, but larger contexts use more memory, so increase it only after measuring. The server listens on port 8080 by default and exposes an OpenAI-compatible API.

NVIDIA recommends temperature 1.0 and top_p 0.95, which you set in each request. The model card’s examples turn reasoning on or off with enable_thinking through chat_template_kwargs. Reasoning tokens and the final answer share the max_tokens budget, so set it high enough.

Step 4: Run test prompts

First, a simple arithmetic check:

bash

curl http://localhost:8080/v1/chat/completions \

  -H “Content-Type: application/json” \

  -d ‘{“messages”:[{“role”:”user”,”content”:”What is 17 multiplied by 23? Reply with the number only.”}],

       “temperature”:1.0,”top_p”:0.95,”max_tokens”:2048}’

The correct answer is 391. If reasoning text appears and cuts off the answer, raise max_tokens.

Second, a structured-output check. Ask for only JSON with the keys “city” and “country” for the Eiffel Tower, then parse the reply. The model card lists JSON output as supported, so the reply should parse without manual cleanup. The model card also lists tool calling, but the llama.cpp commands in NVIDIA’s documentation do not include a verified tool-call parser, so test tool calls with your own client before relying on them.

Step 5: Confirm the model runs on the GPU

While a response is generating, run:

bash

nvidia-smi -l 1

GPU utilization should rise and memory use should be high. If GPU use stays near zero while responses are slow, the model may be running on the CPU. Check the startup log for how many layers were offloaded to the GPU. An out-of-memory error during loading means the weights plus context exceed available VRAM, so reduce -c first.

E4B variant: Download Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf from nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB. NVIDIA’s Quick Start runs llama.cpp in its CUDA container image with –gpus all, a 65536-token context, and the –jinja flag, so it requires Docker and the NVIDIA Container Toolkit. The model summary still labels the llama.cpp recipe as pending, so confirm its current status before relying on it.

How to Benchmark NVIDIA Nemotron 3.5 Bolt

Measure four things:

Follow this procedure:

  1. Record your GPU, driver, system RAM, llama.cpp commit, checkpoint filename, context size, and sampling settings.
  2. Use a fixed prompt set with the same input length for every run.
  3. Set the same max_tokens for every run.
  4. Send one warm-up request and discard its timings.
  5. Repeat each test at least five times and report the median.
  6. Log VRAM with nvidia-smi during each run.
  7. Include a representative task, such as a multi-step lookup or coding request, not only short conversational prompts.

Change one setting at a time, and record the results in a template like this:

MetricResult
GPU and VRAMRecord actual hardware
Model and quantizationRecord exact file
Time to first tokenMedian seconds
Generation speedMedian tokens per second
Peak VRAMMeasured GB
Task qualityPass rate on your task set

This table is a template. It contains no measured values.

NVIDIA’s Official Benchmarks vs. Your Results

NVIDIA’s technical report states a design goal of decoding at least 80 tokens per second for the 74B-A9B model. The report’s hardware and test conditions determine how that figure compares with your system, and I could not confirm a measured speed figure from the sources reviewed, so this guide does not quote one.

NVIDIA’s model card also reports accuracy results measured with its own harness. For the 74B IQ2 build, it lists 86.43 on GPQA Diamond. For E4B at Q4_K_M, it lists 55.60 on MultiChallenge, compared with 39.36 for Gemma 4 E4B-it at the same precision. NVIDIA notes these may differ from other vendors’ self-reported numbers.

These measurements answer different questions. Accuracy scores show correctness on a test set. Generation speed shows how quickly your hardware produces tokens. End-to-end time includes prompt processing, tool calls, and retries. Task quality shows whether the output is correct for your job.

Best Practices for Faster Local Inference

Use the quantization the runtime supports. For the 74B model, NVIDIA ships the IQ2 build. Reduce context length to what your task needs, because the KV cache grows with it. Close other GPU-heavy programs, which compete for VRAM. Keep as much of the model on the GPU as memory allows, and verify this in the startup log.

Speculative decoding can speed generation when the model, drafter, and runtime all work together, but it is not a guaranteed improvement, so benchmark it. Lower precision and shorter context save memory but can reduce output quality. Judge each change by task results, not speed alone.

Common Problems and Troubleshooting

ProblemWhat to check
Out-of-memory errorContext size (-c), other GPU applications, available VRAM
Model fails to loadFile path and name, complete download, llama.cpp version
Slow generationLayers on CPU, GPU use in nvidia-smi, whether CUDA was enabled in the build
Unexpected outputSampling values, max_tokens truncation, reasoning settings
Image input failsWhether the mmproj file is present and passed with –mmproj
Tool calling failsClient request format and runtime support; test with your own client

Limitations and Real-World Use Cases

Local deployment suits developers testing coding or terminal agents, teams evaluating internal tools, and users who want data to stay on hardware they control. Running a model locally does not make your setup private by itself, since logs, plugins, and connected tools can still send data elsewhere. Local hardware also carries upfront cost, electricity, maintenance, and setup effort. Model quality varies by task, so test on your own workload. Review the OpenMDW-1.1 license and your security policies before deployment.

Conclusion

NVIDIA Nemotron 3.5 Bolt is worth testing if you have NVIDIA hardware and a clear workload. Match the variant to your memory budget, verify the checkpoint and runtime on the official model page, and benchmark on your own tasks before using it in anything important. Compare your numbers with NVIDIA’s only when the test conditions match. Your next step is to open the 74B-A9B GGUF repository, download the files, and run the test prompts above.

Frequently Asked Questions

What is NVIDIA Nemotron 3.5 Bolt?

It is an open-weight NVIDIA model family. The 74B-A9B 24GB model handles text, images, and video for agent tasks, while E4B is a text model for conversational assistants.

Can it run on a normal PC?

Only with a suitable NVIDIA GPU is it practical. The 74B-A9B model targets a 24GB memory budget, and E4B targets about 4GB of video memory in its intended workloads.

How much VRAM does it need?

NVIDIA targets under 24GB for the 74B-A9B model at batch size 1 and 128K context, and about 4GB for E4B. Context length, images, and runtime overhead add memory.

What is the difference between 74B-A9B and E4B?

74B-A9B is a larger multimodal mixture-of-experts model for agent work. E4B is an 8-billion-parameter text model based on Gemma 4 E4B for conversation and tool use.

Can you run Nemotron 3.5 Bolt with llama.cpp?

Yes. The 74B-A9B model has an NVIDIA GGUF repository, and E4B has a Quick Start that its page still marks as pending. Check each model page before relying on either path.

How do you measure Nemotron tokens per second?

Divide generated tokens by generation time, excluding prompt processing. Use the same prompt, output length, and warm-up for every run, and report the median of repeated runs.

Is NVIDIA Nemotron 3.5 Bolt free for commercial use?

Both models use the OpenMDW-1.1 license, and their model cards state they are ready for commercial or non-commercial use. Read the full license, since E4B also references Gemma 4’s Apache 2.0 license.

How can you improve local Nemotron inference speed?

Reduce context length, close other GPU programs, confirm the model runs on the GPU, and change one setting at a time. Confirm that any speed gain keeps output quality acceptable.