Skip to content

Commit 658bb6f

Browse files
giswqsCopilotpre-commit-ci[bot]
authored
Add support for SAM3 (#430)
* Add support for SAM3 * Update pyproject.toml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix import error * Move colors import * Fix typehints * Update samgeo/samgeo3.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update samgeo/samgeo3.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update samgeo/samgeo3.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update samgeo/samgeo3.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update samgeo/samgeo3.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update samgeo/samgeo3.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Improve show_anns * Add save_masks method * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add show_masks method * Add generate_masks_by_boxes * Add transformers backedn * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add sam3 notebook example * Add jpg notebook example --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 3deec02 commit 658bb6f

File tree

9 files changed

+1928
-1
lines changed

9 files changed

+1928
-1
lines changed

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,10 @@ private/
1515
**/*.csv
1616
**/*.pt
1717
*.mp4
18+
samgeo/assets/
1819
docs/examples/segments/
1920
docs/examples/blended/
21+
docs/examples/*.png
2022

2123
docs/examples/*.geojson
2224

‎docs/examples/sam3_image_segmentation.ipynb‎

Lines changed: 357 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,357 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# SAM3 Image Segmentation for Remote Sensing\n",
8+
"\n",
9+
"This notebook demonstrates how to use the Segment Anything Model 3 (SAM3) for segmenting remote sensing images using the `samgeo3` module.\n",
10+
"\n",
11+
"## Installation\n",
12+
"\n",
13+
"First, make sure you have the required dependencies installed:"
14+
]
15+
},
16+
{
17+
"cell_type": "code",
18+
"execution_count": null,
19+
"metadata": {},
20+
"outputs": [],
21+
"source": [
22+
"# %pip install \"segment-geospatial[samgeo3]\""
23+
]
24+
},
25+
{
26+
"cell_type": "markdown",
27+
"metadata": {},
28+
"source": [
29+
"## Import Libraries\n"
30+
]
31+
},
32+
{
33+
"cell_type": "code",
34+
"execution_count": null,
35+
"metadata": {},
36+
"outputs": [],
37+
"source": [
38+
"import leafmap\n",
39+
"from samgeo import SamGeo3, download_file"
40+
]
41+
},
42+
{
43+
"cell_type": "markdown",
44+
"metadata": {},
45+
"source": [
46+
"## Download Sample Data\n",
47+
"\n",
48+
"Let's download a sample satellite image covering the University of California, Berkeley, for testing:\n"
49+
]
50+
},
51+
{
52+
"cell_type": "code",
53+
"execution_count": null,
54+
"metadata": {},
55+
"outputs": [],
56+
"source": [
57+
"url = \"https://huggingface.co/datasets/giswqs/geospatial/resolve/main/uc_berkeley.tif\"\n",
58+
"image_path = download_file(url)"
59+
]
60+
},
61+
{
62+
"cell_type": "code",
63+
"execution_count": null,
64+
"metadata": {},
65+
"outputs": [],
66+
"source": [
67+
"m = leafmap.Map()\n",
68+
"m.add_raster(image_path, layer_name=\"Satellite image\")\n",
69+
"m"
70+
]
71+
},
72+
{
73+
"cell_type": "markdown",
74+
"metadata": {},
75+
"source": [
76+
"## Request access to SAM3\n",
77+
"\n",
78+
"To use SAM3, you need to request access by filling out this form on Hugging Face: https://huggingface.co/facebook/sam3\n",
79+
"\n",
80+
"Once you have access, uncomment the following code block and run it."
81+
]
82+
},
83+
{
84+
"cell_type": "code",
85+
"execution_count": null,
86+
"metadata": {},
87+
"outputs": [],
88+
"source": [
89+
"# from huggingface_hub import login\n",
90+
"# login()"
91+
]
92+
},
93+
{
94+
"cell_type": "markdown",
95+
"metadata": {},
96+
"source": [
97+
"## Initialize SAM3\n",
98+
"\n",
99+
"When initializing SAM3, you can choose the backend from \"meta\", or \"transformers\"."
100+
]
101+
},
102+
{
103+
"cell_type": "code",
104+
"execution_count": null,
105+
"metadata": {},
106+
"outputs": [],
107+
"source": [
108+
"sam3 = SamGeo3(backend=\"meta\", device=None, checkpoint_path=None, load_from_HF=True)"
109+
]
110+
},
111+
{
112+
"cell_type": "markdown",
113+
"metadata": {},
114+
"source": [
115+
"## Set the image\n",
116+
"\n",
117+
"You can set the image by either passing the image path or the image URL."
118+
]
119+
},
120+
{
121+
"cell_type": "code",
122+
"execution_count": null,
123+
"metadata": {},
124+
"outputs": [],
125+
"source": [
126+
"sam3.set_image(image_path)"
127+
]
128+
},
129+
{
130+
"cell_type": "markdown",
131+
"metadata": {},
132+
"source": [
133+
"## Generate masks with text prompt"
134+
]
135+
},
136+
{
137+
"cell_type": "code",
138+
"execution_count": null,
139+
"metadata": {},
140+
"outputs": [],
141+
"source": [
142+
"sam3.generate_masks(prompt=\"building\")"
143+
]
144+
},
145+
{
146+
"cell_type": "markdown",
147+
"metadata": {},
148+
"source": [
149+
"## Show the results"
150+
]
151+
},
152+
{
153+
"cell_type": "code",
154+
"execution_count": null,
155+
"metadata": {},
156+
"outputs": [],
157+
"source": [
158+
"sam3.show_anns()"
159+
]
160+
},
161+
{
162+
"cell_type": "markdown",
163+
"metadata": {},
164+
"source": [
165+
"![annotation](https://github.com/user-attachments/assets/64323223-35a2-4e03-9cee-1b60fa0c12af)"
166+
]
167+
},
168+
{
169+
"cell_type": "code",
170+
"execution_count": null,
171+
"metadata": {},
172+
"outputs": [],
173+
"source": [
174+
"sam3.show_masks()"
175+
]
176+
},
177+
{
178+
"cell_type": "markdown",
179+
"metadata": {},
180+
"source": [
181+
"## Generate masks by bounding boxes"
182+
]
183+
},
184+
{
185+
"cell_type": "code",
186+
"execution_count": null,
187+
"metadata": {},
188+
"outputs": [],
189+
"source": [
190+
"# Define boxes in [xmin, ymin, xmax, ymax] format\n",
191+
"boxes = [[-122.2597, 37.8709, -122.2587, 37.8717]]\n",
192+
"\n",
193+
"# Optional: specify which boxes are positive/negative prompts\n",
194+
"box_labels = [True] # True=include, False=exclude\n",
195+
"\n",
196+
"# Generate masks\n",
197+
"sam3.generate_masks_by_boxes(boxes, box_labels, box_crs=\"EPSG:4326\")"
198+
]
199+
},
200+
{
201+
"cell_type": "code",
202+
"execution_count": null,
203+
"metadata": {},
204+
"outputs": [],
205+
"source": [
206+
"sam3.show_boxes(boxes, box_labels, box_crs=\"EPSG:4326\")"
207+
]
208+
},
209+
{
210+
"cell_type": "markdown",
211+
"metadata": {},
212+
"source": [
213+
"![bbox](https://github.com/user-attachments/assets/87f0b9ab-a621-4e08-a4de-c90cbf0bf32f)"
214+
]
215+
},
216+
{
217+
"cell_type": "markdown",
218+
"metadata": {},
219+
"source": [
220+
"## Show the results"
221+
]
222+
},
223+
{
224+
"cell_type": "code",
225+
"execution_count": null,
226+
"metadata": {},
227+
"outputs": [],
228+
"source": [
229+
"sam3.show_anns()"
230+
]
231+
},
232+
{
233+
"cell_type": "markdown",
234+
"metadata": {},
235+
"source": [
236+
"## Save Masks\n",
237+
"\n",
238+
"Save the generated masks to a file. If the input is a GeoTIFF, the output will be a GeoTIFF with the same georeferencing. Otherwise, it will be saved as PNG."
239+
]
240+
},
241+
{
242+
"cell_type": "code",
243+
"execution_count": null,
244+
"metadata": {},
245+
"outputs": [],
246+
"source": [
247+
"# Save masks with unique values for each object\n",
248+
"# Since satellite.tif is a GeoTIFF, the output will also be a GeoTIFF\n",
249+
"sam3.save_masks(output=\"building_masks.tif\", unique=True)"
250+
]
251+
},
252+
{
253+
"cell_type": "code",
254+
"execution_count": null,
255+
"metadata": {},
256+
"outputs": [],
257+
"source": [
258+
"# Save as binary mask (all foreground pixels are 255)\n",
259+
"sam3.save_masks(output=\"building_masks_binary.tif\", unique=False)"
260+
]
261+
},
262+
{
263+
"cell_type": "markdown",
264+
"metadata": {},
265+
"source": [
266+
"## Save Masks with Confidence Scores\n",
267+
"\n",
268+
"You can also save the confidence scores for each mask. The scores indicate the model's confidence for each predicted mask."
269+
]
270+
},
271+
{
272+
"cell_type": "code",
273+
"execution_count": null,
274+
"metadata": {},
275+
"outputs": [],
276+
"source": [
277+
"# Save masks and confidence scores\n",
278+
"# Each pixel in the scores image will have the confidence value of its mask\n",
279+
"sam3.save_masks(\n",
280+
" output=\"building_masks_with_scores.tif\",\n",
281+
" save_scores=\"building_scores.tif\",\n",
282+
" unique=True,\n",
283+
")"
284+
]
285+
},
286+
{
287+
"cell_type": "code",
288+
"execution_count": null,
289+
"metadata": {},
290+
"outputs": [],
291+
"source": [
292+
"sam3.show_masks(cmap=\"coolwarm\")"
293+
]
294+
},
295+
{
296+
"cell_type": "markdown",
297+
"metadata": {},
298+
"source": [
299+
"![scores](https://github.com/user-attachments/assets/23ec9b07-0de9-4f72-81b2-83a3c499e94e)"
300+
]
301+
},
302+
{
303+
"cell_type": "markdown",
304+
"metadata": {},
305+
"source": [
306+
"### Visualize Confidence Scores\n",
307+
"\n",
308+
"Let's visualize the confidence scores to see which masks have higher confidence:"
309+
]
310+
},
311+
{
312+
"cell_type": "code",
313+
"execution_count": null,
314+
"metadata": {},
315+
"outputs": [],
316+
"source": [
317+
"m.add_raster(\"building_masks.tif\", layer_name=\"Building masks\", visible=False)\n",
318+
"m.add_raster(\n",
319+
" \"building_scores.tif\",\n",
320+
" layer_name=\"Building scores\",\n",
321+
" cmap=\"coolwarm\",\n",
322+
" opacity=0.8,\n",
323+
" nodata=0,\n",
324+
")\n",
325+
"m"
326+
]
327+
},
328+
{
329+
"cell_type": "markdown",
330+
"metadata": {},
331+
"source": [
332+
"![map](https://github.com/user-attachments/assets/fa21320d-b4f3-48f9-a3f2-828f4ed1c567)"
333+
]
334+
}
335+
],
336+
"metadata": {
337+
"kernelspec": {
338+
"display_name": "geo",
339+
"language": "python",
340+
"name": "python3"
341+
},
342+
"language_info": {
343+
"codemirror_mode": {
344+
"name": "ipython",
345+
"version": 3
346+
},
347+
"file_extension": ".py",
348+
"mimetype": "text/x-python",
349+
"name": "python",
350+
"nbconvert_exporter": "python",
351+
"pygments_lexer": "ipython3",
352+
"version": "3.12.2"
353+
}
354+
},
355+
"nbformat": 4,
356+
"nbformat_minor": 2
357+
}

0 commit comments

Comments
 (0)