From 2c572cff7b281c27c99446db3e7dd0b59afdbe2f Mon Sep 17 00:00:00 2001 From: Wenchen Fan Date: Sat, 27 Dec 2025 23:06:36 +0800 Subject: [PATCH 1/4] Introduce Cursor/VS Code support --- developer-tools.md | 19 +++++++++++++++++++ site/developer-tools.html | 22 ++++++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/developer-tools.md b/developer-tools.md index 0908cef343..6595426f02 100644 --- a/developer-tools.md +++ b/developer-tools.md @@ -470,6 +470,25 @@ debug in IntelliJ as usual. To exit remote debug mode (so that you don't have to keep starting the remote debugger), type "session clear" in SBT console while you're in a project. + +

Cursor/VS Code

+ +You can leverage the power of *Cursor* or *VS Code* for Spark development by using the Metals extension. +Metals provides robust Scala support through the Build Server Protocol (BSP). + +Initial setup: +- *Install Metals*: Search for and install the Scala (Metals) extension from the Marketplace. +- *Open Project*: Open your Spark root directory. +- *Select Build Tool*: When prompted by the popup in the bottom-left corner, select sbt as your build tool. +- *Confirm Build Server*: Metals defaults to using Bloop as the build server. Do not change this setting, as Bloop provides the fastest compilation and best integration for local development. Run `Metals: Run doctor` in the Command Palette to confirm it. + +If you encounter compilation errors or the IDE fails to recognize symbols, follow these steps to perform a clean import: +1. *Clean Workspace*: Run the following command in your terminal to remove cached build metadata: `rm -rf .bloop/ .metals/ .bsp/`. +2. *Reload Window*: Open the Command Palette (Cmd+Shift+P or Ctrl+Shift+P), run `Developer: Reload Window`, and select sbt when the prompt reappears. +3. *Import Build*: Run `Metals: Import build` from the Command Palette and wait for the process to complete. +4. *Connect to Server*: Run `Metals: Connect to build server` and wait for the process to complete. +5. *Verify Health*: Run `Metals: Run doctor`. Ensure the status dashboard shows: `Build definition is coming from sbt` and `Build server currently being used is Bloop`. +

Eclipse

Eclipse can be used to develop and test Spark. The following configuration is known to work: diff --git a/site/developer-tools.html b/site/developer-tools.html index 574ef1f201..3cba1212ad 100644 --- a/site/developer-tools.html +++ b/site/developer-tools.html @@ -606,6 +606,28 @@
Trigger the remote debugging

To exit remote debug mode (so that you don’t have to keep starting the remote debugger), type “session clear” in SBT console while you’re in a project.

+

Cursor/VS Code

+ +

You can leverage the power of Cursor or VS Code for Spark development by using the Metals extension. +Metals provides robust Scala support through the Build Server Protocol (BSP).

+ +

Initial setup:

+ + +

If you encounter compilation errors or the IDE fails to recognize symbols, follow these steps to perform a clean import:

+
    +
  1. Clean Workspace: Run the following command in your terminal to remove cached build metadata: rm -rf .bloop/ .metals/ .bsp/.
    2. +
  2. Reload Window: Open the Command Palette (Cmd+Shift+P or Ctrl+Shift+P), run Developer: Reload Window, and select sbt when the prompt reappears.
    3. +
  3. Import Build: Run Metals: Import build from the Command Palette and wait for the process to complete.
    4. +
  4. Connect to Server: Run Metals: Connect to build server and wait for the process to complete.
    5. +
  5. Verify Health: Run Metals: Run doctor. Ensure the status dashboard shows: Build definition is coming from sbt and Build server currently being used is Bloop.
    6. +
+

Eclipse

Eclipse can be used to develop and test Spark. The following configuration is known to work:

From e7b197da6ee4f399604bbcae36bb3ae411fcc848 Mon Sep 17 00:00:00 2001 From: Wenchen Fan Date: Sat, 27 Dec 2025 23:45:41 +0800 Subject: [PATCH 2/4] improve --- developer-tools.md | 8 ++++---- site/developer-tools.html | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/developer-tools.md b/developer-tools.md index 6595426f02..b6120b1533 100644 --- a/developer-tools.md +++ b/developer-tools.md @@ -480,14 +480,14 @@ Initial setup: - *Install Metals*: Search for and install the Scala (Metals) extension from the Marketplace. - *Open Project*: Open your Spark root directory. - *Select Build Tool*: When prompted by the popup in the bottom-left corner, select sbt as your build tool. -- *Confirm Build Server*: Metals defaults to using Bloop as the build server. Do not change this setting, as Bloop provides the fastest compilation and best integration for local development. Run `Metals: Run doctor` in the Command Palette to confirm it. +- *Confirm Build Server*: Metals defaults to using Bloop as the build server. Do not change this setting, as Bloop provides the fastest compilation and best integration for local development. Open the Command Palette (Cmd+Shift+P or Ctrl+Shift+P), run `Metals: Run doctor`, and verify the status dashboard. Ensure it shows: `Build definition is coming from sbt` and `Build server currently being used is Bloop`. If you encounter compilation errors or the IDE fails to recognize symbols, follow these steps to perform a clean import: 1. *Clean Workspace*: Run the following command in your terminal to remove cached build metadata: `rm -rf .bloop/ .metals/ .bsp/`. -2. *Reload Window*: Open the Command Palette (Cmd+Shift+P or Ctrl+Shift+P), run `Developer: Reload Window`, and select sbt when the prompt reappears. -3. *Import Build*: Run `Metals: Import build` from the Command Palette and wait for the process to complete. +2. *Reload Window*: Run `Developer: Reload Window` from the Command Palette, and select sbt when the prompt reappears. +3. *Import Build*: Run `Metals: Import build` and wait for the process to complete. 4. *Connect to Server*: Run `Metals: Connect to build server` and wait for the process to complete. -5. *Verify Health*: Run `Metals: Run doctor`. Ensure the status dashboard shows: `Build definition is coming from sbt` and `Build server currently being used is Bloop`. +5. *Confirm*: Run `Metals: Run doctor` and verify the status dashboard.

Eclipse

diff --git a/site/developer-tools.html b/site/developer-tools.html index 3cba1212ad..98794dde57 100644 --- a/site/developer-tools.html +++ b/site/developer-tools.html @@ -616,16 +616,16 @@

Cursor/VS Code

  • Install Metals: Search for and install the Scala (Metals) extension from the Marketplace.
  • Open Project: Open your Spark root directory.
  • Select Build Tool: When prompted by the popup in the bottom-left corner, select sbt as your build tool.
    • -
  • Confirm Build Server: Metals defaults to using Bloop as the build server. Do not change this setting, as Bloop provides the fastest compilation and best integration for local development. Run Metals: Run doctor in the Command Palette to confirm it.
    • +
  • Confirm Build Server: Metals defaults to using Bloop as the build server. Do not change this setting, as Bloop provides the fastest compilation and best integration for local development. Open the Command Palette (Cmd+Shift+P or Ctrl+Shift+P), run Metals: Run doctor, and verify the status dashboard. Ensure it shows: Build definition is coming from sbt and Build server currently being used is Bloop.

    • If you encounter compilation errors or the IDE fails to recognize symbols, follow these steps to perform a clean import:

    1. Clean Workspace: Run the following command in your terminal to remove cached build metadata: rm -rf .bloop/ .metals/ .bsp/.
      2. -
    2. Reload Window: Open the Command Palette (Cmd+Shift+P or Ctrl+Shift+P), run Developer: Reload Window, and select sbt when the prompt reappears.
      3. -
    3. Import Build: Run Metals: Import build from the Command Palette and wait for the process to complete.
      4. +
    4. Reload Window: Run Developer: Reload Window from the Command Palette, and select sbt when the prompt reappears.
      5. +
    5. Import Build: Run Metals: Import build and wait for the process to complete.
    6. Connect to Server: Run Metals: Connect to build server and wait for the process to complete.
      7. -
    7. Verify Health: Run Metals: Run doctor. Ensure the status dashboard shows: Build definition is coming from sbt and Build server currently being used is Bloop.
      8. +
    8. Confirm: Run Metals: Run doctor and verify the status dashboard.

    Eclipse

    From 12092cd0803cdb3b49fc8da9bc40311f1ef71bbc Mon Sep 17 00:00:00 2001 From: Wenchen Fan Date: Sun, 28 Dec 2025 10:28:33 +0800 Subject: [PATCH 3/4] Apply suggestions from code review --- developer-tools.md | 2 +- site/developer-tools.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/developer-tools.md b/developer-tools.md index b6120b1533..6025b06494 100644 --- a/developer-tools.md +++ b/developer-tools.md @@ -479,7 +479,7 @@ Metals provides robust Scala support through the Build Server Protocol (BSP). Initial setup: - *Install Metals*: Search for and install the Scala (Metals) extension from the Marketplace. - *Open Project*: Open your Spark root directory. -- *Select Build Tool*: When prompted by the popup in the bottom-left corner, select sbt as your build tool. +- *Select Build Tool*: When prompted by the popup in the bottom-left corner, select sbt to read build definition. - *Confirm Build Server*: Metals defaults to using Bloop as the build server. Do not change this setting, as Bloop provides the fastest compilation and best integration for local development. Open the Command Palette (Cmd+Shift+P or Ctrl+Shift+P), run `Metals: Run doctor`, and verify the status dashboard. Ensure it shows: `Build definition is coming from sbt` and `Build server currently being used is Bloop`. If you encounter compilation errors or the IDE fails to recognize symbols, follow these steps to perform a clean import: diff --git a/site/developer-tools.html b/site/developer-tools.html index 98794dde57..7c1eee72cb 100644 --- a/site/developer-tools.html +++ b/site/developer-tools.html @@ -615,7 +615,7 @@

    Cursor/VS Code

    • Install Metals: Search for and install the Scala (Metals) extension from the Marketplace.
    • Open Project: Open your Spark root directory.
      • -
    • Select Build Tool: When prompted by the popup in the bottom-left corner, select sbt as your build tool.
      • +
    • Select Build Tool: When prompted by the popup in the bottom-left corner, select sbt to read build definition.
    • Confirm Build Server: Metals defaults to using Bloop as the build server. Do not change this setting, as Bloop provides the fastest compilation and best integration for local development. Open the Command Palette (Cmd+Shift+P or Ctrl+Shift+P), run Metals: Run doctor, and verify the status dashboard. Ensure it shows: Build definition is coming from sbt and Build server currently being used is Bloop.
    From 5fc9a7d0e3a4412aea87a8036eab971c6399e60d Mon Sep 17 00:00:00 2001 From: Wenchen Fan Date: Sun, 28 Dec 2025 20:01:49 +0800 Subject: [PATCH 4/4] Apply suggestions from code review --- developer-tools.md | 11 ++++++----- site/developer-tools.html | 1 + 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/developer-tools.md b/developer-tools.md index 6025b06494..bf0920a6e2 100644 --- a/developer-tools.md +++ b/developer-tools.md @@ -483,11 +483,12 @@ Initial setup: - *Confirm Build Server*: Metals defaults to using Bloop as the build server. Do not change this setting, as Bloop provides the fastest compilation and best integration for local development. Open the Command Palette (Cmd+Shift+P or Ctrl+Shift+P), run `Metals: Run doctor`, and verify the status dashboard. Ensure it shows: `Build definition is coming from sbt` and `Build server currently being used is Bloop`. If you encounter compilation errors or the IDE fails to recognize symbols, follow these steps to perform a clean import: -1. *Clean Workspace*: Run the following command in your terminal to remove cached build metadata: `rm -rf .bloop/ .metals/ .bsp/`. -2. *Reload Window*: Run `Developer: Reload Window` from the Command Palette, and select sbt when the prompt reappears. -3. *Import Build*: Run `Metals: Import build` and wait for the process to complete. -4. *Connect to Server*: Run `Metals: Connect to build server` and wait for the process to complete. -5. *Confirm*: Run `Metals: Run doctor` and verify the status dashboard. +1. *Stop Bloop Server*: Run the following command in your terminal to stop the bloop server: `pkill -f "bloop.BloopServer"`. If you have installed bloop CLI you can run `bloop exit` instead. +2. *Clean Workspace*: Run the following command in your terminal to remove cached build metadata: `rm -rf .bloop/ .metals/ .bsp/`. +3. *Reload Window*: Run `Developer: Reload Window` from the Command Palette, and select sbt when the prompt reappears. +4. *Import Build*: Run `Metals: Import build` and wait for the process to complete. +5. *Connect to Server*: Run `Metals: Connect to build server` and wait for the process to complete. +6. *Confirm*: Run `Metals: Run doctor` and verify the status dashboard.

    Eclipse

    diff --git a/site/developer-tools.html b/site/developer-tools.html index 7c1eee72cb..ea160cfb23 100644 --- a/site/developer-tools.html +++ b/site/developer-tools.html @@ -621,6 +621,7 @@

    Cursor/VS Code

    If you encounter compilation errors or the IDE fails to recognize symbols, follow these steps to perform a clean import:

      +
    1. Stop Bloop Server: Run the following command in your terminal to stop the bloop server: pkill -f "bloop.BloopServer". If you have installed bloop CLI you can run bloop exit instead.
    2. Clean Workspace: Run the following command in your terminal to remove cached build metadata: rm -rf .bloop/ .metals/ .bsp/.
    3. Reload Window: Run Developer: Reload Window from the Command Palette, and select sbt when the prompt reappears.
    4. Import Build: Run Metals: Import build and wait for the process to complete.