How to segment objects with SAM 3 using text prompts in Python

thijs · September 2, 2026, 12:18am

Based on contributions by Tim.

SAM 3 is Meta’s open-vocabulary segmentation model. Instead of clicking on an object or drawing a bounding box, you describe what you’re looking for in plain text, "red cylinder", "gripper", "white coffee mug", and SAM 3 returns a segmentation mask and bounding box for every matching instance it finds in the image.

By the end of this how-to you’ll have a working Python script that loads an image, runs SAM 3 with a text prompt, and prints the bounding box and confidence score of each detected object.

What you need

Hardware

A CUDA-compatible NVIDIA GPU. SAM 3 is not practical on CPU.
At least 8 GB VRAM recommended; below that you will likely hit out-of-memory errors at default resolution.
CUDA 12.6 or higher.
A reasonably modern GPU (RTX 3060 or better is a comfortable baseline).

Operating system

SAM 3 is developed and tested on Linux. If you’re on Windows, run everything inside WSL (Windows Subsystem for Linux) with Ubuntu — do not try to install it natively on Windows, the PyTorch CUDA builds and the SAM 3 package expect a Linux environment. All commands below assume a WSL Ubuntu (or native Linux) terminal.

Software

Python 3.12 or higher
Git
A Hugging Face account with access to the SAM 3 checkpoint (see step 1)

Steps

1. Request access to the model checkpoint

SAM 3’s weights are hosted on Hugging Face and are gated — you need to request access before you can download them.

Go to https://huggingface.co/facebook/sam3
Log in with your Hugging Face account (create one for free if you don’t have one)
Click Request access and accept the license terms
Access is typically granted within minutes to a few hours

Once approved, generate an access token:

Go to https://huggingface.co/settings/tokens
Click New token, give it a name, select the Read role, and copy the token

You’ll need this token in step 4.

2. Create a conda environment

conda create -n sam3 python=3.12
conda activate sam3

Verify the correct Python version is active:

python --version
# Should output: Python 3.12.x

3. Install PyTorch with CUDA support

pip install torch==2.7.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126

Verify PyTorch can see your GPU:

python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"

If cuda.is_available() returns False, your CUDA drivers aren’t installed or aren’t visible inside WSL. Fix this before continuing — SAM 3 will not run without GPU access.

4. Authenticate with Hugging Face

pip install huggingface_hub
hf auth login

Paste your access token when prompted. This stores credentials locally so SAM 3 can automatically download the checkpoint on first run.

5. Clone and install SAM 3

git clone https://github.com/facebookresearch/sam3.git
cd sam3
pip install -e .

The -e flag installs the package in editable mode: changes to the cloned source take effect immediately without reinstalling, useful if you want to inspect or modify the source later.

Verify the installation:

python -c "from sam3.model_builder import build_sam3_image_model; print('SAM 3 installed successfully')"

6. Run your first detection

Create detect.py. Replace "your_image.jpg" with the path to an image and "your object" with a description of something visible in it.

import torch
from PIL import Image
from sam3.model_builder import build_sam3_image_model
from sam3.model.sam3_image_processor import Sam3Processor

# Load the model — this downloads the checkpoint on first run (~1.7 GB)
# and takes 10-30 seconds depending on your hardware.
model = build_sam3_image_model()
processor = Sam3Processor(model)

# Load an image
image = Image.open("your_image.jpg")

# Pass the image to the processor
inference_state = processor.set_image(image)

# Run detection with a text prompt
output = processor.set_text_prompt(state=inference_state, prompt="your object")

# Unpack results
masks = output["masks"]    # binary segmentation masks — shape: (N, H, W)
boxes = output["boxes"]    # bounding boxes — shape: (N, 4), [x_min, y_min, x_max, y_max]
scores = output["scores"]  # confidence scores — shape: (N,)

if len(scores) == 0:
    print("No objects detected. Try a different prompt or check the image.")
else:
    print(f"Found {len(scores)} instance(s).")
    for i, (box, score) in enumerate(zip(boxes, scores)):
        print(f"  Instance {i + 1}: score={score:.2f}, box={box.tolist()}")

Run it:

python detect.py

If the object is present, you’ll see something like:

Found 2 instance(s).
  Instance 1: score=0.91, box=[142.3, 88.7, 310.5, 412.1]
  Instance 2: score=0.73, box=[502.1, 201.3, 680.4, 455.8]

Each box is in pixel coordinates [x_min, y_min, x_max, y_max]. The masks array has one binary mask per instance, at the same resolution as the input image; a True value means that pixel belongs to the detected object.

On first run, SAM 3 downloads the checkpoint from Hugging Face (~1.7 GB) — a one-time cost. Subsequent runs load from cache and start in 10–30 seconds.

7. Filter by confidence

In practice, ignore low-confidence detections before acting on the results:

THRESHOLD = 0.5

valid_results = [
    (mask, box, score)
    for mask, box, score in zip(masks, boxes, scores)
    if score > THRESHOLD
]

if not valid_results:
    print("No confident detections above threshold.")
else:
    best_mask, best_box, best_score = max(valid_results, key=lambda x: x[2])
    print(f"Best detection: score={best_score:.2f}, box={best_box.tolist()}")

0.5 is a reasonable starting point. Lower it if you’re missing real objects, raise it if you’re getting false positives.

Tips for good results

Be specific in your prompt. "blue plastic cube" works better than "object". "metallic cylindrical container" outperforms "thing on the table". Match the prompt to what the camera can clearly distinguish.
Prompt for what is visually distinct. SAM 3 works from visual features. If two objects look nearly identical, differentiate them by color, material, or position — "cube on the left", "red one".
Don’t run it on every frame. SAM 3 takes 1–5 seconds per image depending on your GPU; it is not a real-time detector. Trigger inference only when needed (for example, once at the start of a task to locate a target), not continuously on a video stream.
GPU memory. The model uses roughly 4–6 GB VRAM at typical image resolutions. If you hit out-of-memory errors, reduce the input resolution before passing it to processor.set_image():
```
image = image.resize((640, 480))
```

Troubleshooting

401 error on model download: your hf auth login credentials are missing or expired — re-run hf auth login.
cuda.is_available() is False: your CUDA drivers aren’t installed or aren’t visible from inside WSL; fix this before anything else, SAM 3 will not run on CPU.