Deepseek Coder not working

[rhusiev]

When trying to use deepseek coder (via ollama) and its tokenizer and tokens for fim, the result seems completely irrelevant (or, maybe, cut off). However, when using the prompt I would expect to go to the model directly in the ollama, everything works fine:

vs

Here is my config for llm.nvim:

require("llm").setup({
    model = "deepseek-coder:1.3b-base",
	enable_suggestions_on_startup = true,
	accept_keymap = "<C-M-j>",
	dismiss_keymap = "<C-M-k>",
	tokens_to_clear = {
        "<|endoftext|>",
    },
	fim = {
		enabled = true,
                prefix = "<｜fim▁begin｜>",
                middle = "<｜fim▁hole｜>",
                suffix = "<｜fim▁end｜>"
	},
	backend = "ollama",
	debounce_ms = 0,
	url = "http://localhost:11434/api/generate",
	context_window = 240,
	-- cf https://github.com/ollama/ollama/blob/main/docs/api.md#parameters
	request_body = {
		-- Modelfile options for the model you use
		options = {
			num_predict = 4,
			temperature = 0.2,
			top_p = 0.95,
		},
	},
	lsp = {
		bin_path = vim.api.nvim_call_function("stdpath", { "data" }) .. "/mason/bin/llm-ls",
	},
	tokenizer = {
                repository = "deepseek-ai/deepseek-vl-1.3b-base", -- not working for some reason
	},
})

I believe it is a problem with how llm-ls handles it, but if I am wrong, I will open an issue on the llm.nvim github

[ytyt-yt]

@rhusiev you can try:

prefix = "<｜fim▁begin｜>",
middle = "<｜fim▁end｜>",
suffix = "<｜fim▁hole｜>",

It's working on my end (deepseek-coder-v2)

[gitsang]

prefix = "<｜fim▁begin｜>",
middle = "<｜fim▁end｜>",
suffix = "<｜fim▁hole｜>",

not work for me (deepseek-coder-v2:16b):

require('llm').setup({
  backend = "ollama", -- backend ID, "huggingface" | "ollama" | "openai" | "tgi"
  url = "http://127.0.0.1:11434", -- the http url of the backend
  model = "deepseek-coder-v2:16b", -- the model ID, behavior depends on backend
  api_token = nil, -- cf Install paragraph
  tokens_to_clear = { "<|endoftext|>" }, -- tokens to remove from the model's output
  -- parameters that are added to the request body, values are arbitrary, you can set any field:value pair here it will be passed as is to the backend
  request_body = {
    parameters = {
      max_new_tokens = 6000,
      temperature = 0.2,
      top_p = 0.95,
    },
  },
  -- set this if the model supports fill in the middle
  fim = {
    enabled = true,
    prefix = "<｜fim▁begin｜>",
    middle = "<｜fim▁end｜>",
    suffix = "<｜fim▁hole｜>",
  },
  debounce_ms = 150,
  accept_keymap = "<C-g>",
  dismiss_keymap = "<S-Tab>",
  tls_skip_verify_insecure = false,
  -- llm-ls configuration, cf llm-ls section
  lsp = {
    bin_path = "/home/debian/.local/src/llm-ls/target/release/llm-ls",
    host = nil,
    port = nil,
    cmd_env = { LLM_LOG_LEVEL = "DEBUG" }, -- or { LLM_LOG_LEVEL = "DEBUG" } to set the log level of llm-ls
    version = "0.5.3",
  },
  tokenizer = nil, -- cf Tokenizer paragraph
  context_window = 4096, -- max number of tokens for the context window
  enable_suggestions_on_startup = true,
  enable_suggestions_on_files = "*", -- pattern matching syntax to enable suggestions on specific files, either a string or a list of strings
  disable_url_path_completion = false, -- cf Backend
})
EOF

[ytyt-yt]

prefix = "<｜fim▁begin｜>",
middle = "<｜fim▁end｜>",
suffix = "<｜fim▁hole｜>",

not work for me (deepseek-coder-v2:16b):

require('llm').setup({
  backend = "ollama", -- backend ID, "huggingface" | "ollama" | "openai" | "tgi"
  url = "http://127.0.0.1:11434", -- the http url of the backend
  model = "deepseek-coder-v2:16b", -- the model ID, behavior depends on backend
  api_token = nil, -- cf Install paragraph
  tokens_to_clear = { "<|endoftext|>" }, -- tokens to remove from the model's output
  -- parameters that are added to the request body, values are arbitrary, you can set any field:value pair here it will be passed as is to the backend
  request_body = {
    parameters = {
      max_new_tokens = 6000,
      temperature = 0.2,
      top_p = 0.95,
    },
  },
  -- set this if the model supports fill in the middle
  fim = {
    enabled = true,
    prefix = "<｜fim▁begin｜>",
    middle = "<｜fim▁end｜>",
    suffix = "<｜fim▁hole｜>",
  },
  debounce_ms = 150,
  accept_keymap = "<C-g>",
  dismiss_keymap = "<S-Tab>",
  tls_skip_verify_insecure = false,
  -- llm-ls configuration, cf llm-ls section
  lsp = {
    bin_path = "/home/debian/.local/src/llm-ls/target/release/llm-ls",
    host = nil,
    port = nil,
    cmd_env = { LLM_LOG_LEVEL = "DEBUG" }, -- or { LLM_LOG_LEVEL = "DEBUG" } to set the log level of llm-ls
    version = "0.5.3",
  },
  tokenizer = nil, -- cf Tokenizer paragraph
  context_window = 4096, -- max number of tokens for the context window
  enable_suggestions_on_startup = true,
  enable_suggestions_on_files = "*", -- pattern matching syntax to enable suggestions on specific files, either a string or a list of strings
  disable_url_path_completion = false, -- cf Backend
})
EOF

I think you can try a base model (such as deepseek-coder-v2:16b-lite-base-q4_0) instead of an instruct model

[gitsang]

I think you can try a base model (such as deepseek-coder-v2:16b-lite-base-q4_0) instead of an instruct model

I eventually found that in the scenario of using ollama, some changes need to be made to the configuration: #111 (comment)