2 BUGS and 1 possible enhancement. #314
Description
DETAILS:
KToolBox version 0.18.2.
System is Windows.
This is my .env file:
KTOOLBOX_DOWNLOADER__RETRY_TIMES=2
KTOOLBOX_JOB__COUNT=1
KTOOLBOX_JOB__INCLUDE_REVISIONS=True
KTOOLBOX_JOB__POST_DIRNAME_FORMAT={id} {title} {published} {user} {service}
KTOOLBOX_JOB__POST_STRUCTURE__ATTACHMENTS=./
KTOOLBOX_JOB__FILENAME_FORMAT=[{title}]_{}
KTOOLBOX_JOB__SEQUENTIAL_FILENAME=True
KTOOLBOX_JOB__SEQUENTIAL_FILENAME_EXCLUDES='[".psd", ".zip", ".mp4"]'
KTOOLBOX_JOB__EXTRACT_EXTERNAL_LINKS=True
KTOOLBOX_JOB__EXTERNAL_LINKS=./BUG 1:
When I use the next command inside of my .env file "KTOOLBOX_JOB__INCLUDE_REVISIONS=True" combined with the previous .env commands, I SOMETIMES get the next errors one or several times in a row:
This artist gives the error 8 times in a row = https://kemono.su/patreon/user/19931820
This artist does NOT give the error: = https://kemono.cr/patreon/user/18969333
2025-08-09 21:07:00 | ERROR | ktoolbox.api.base - Kemono API call failed - {'ret': APIRet(code=1002, message="1 validation error for Response\n Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='<!DOCTYPE html><html lan...</script></body></html>', input_type=str]\n For further information visit https://errors.pydantic.dev/2.11/v/json_invalid", exception=1 validation error for Response
Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='<!DOCTYPE html><html lan...</script></body></html>', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/json_invalid, data=None)}BUG 2:
When I add the next command inside of my .env file:
KTOOLBOX_JOB__EXTERNAL_LINK_PATTERNS='["https?://drive\.google\.com/[^\\s]+", "https?://docs\.google\.com/[^\\s]+", "https?://mega\.nz/[^\\s]+", "https?://mega\.co\.nz/[^\\s]+", "https?://(?:www\.)?dropbox\.com/[^\\s]+", "https?://db\.tt/[^\\s]+", "https?://onedrive\.live\.com/[^\\s]+", "https?://1drv\.ms/[^\\s]+", "https?://(?:www\.)?mediafire\.com/[^\\s]+", "https?://(?:www\.)?wetransfer\.com/[^\\s]+", "https?://we\.tl/[^\\s]+", "https?://(?:www\.)?sendspace\.com/[^\\s]+", "https?://(?:www\.)?4shared\.com/[^\\s]+", "https?://(?:www\.)?zippyshare\.com/[^\\s]+", "https?://(?:www\.)?uploadfiles\.io/[^\\s]+", "https?://(?:www\.)?box\.com/[^\\s]+", "https?://(?:www\.)?pcloud\.com/[^\\s]+", "https?://disk\.yandex\.[a-z]+/[^\\s]+", "https?://[^\\s](?:file|upload|share|download|drive|storage)[^\\s]\.[a-z]{2,4}/[^\\s]+"]'I get this series of errors:
Traceback (most recent call last):
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\pydantic_settings\sources\base.py", line 490, in __call__
field_value = self.prepare_field_value(field_name, field, field_value, value_is_complex)
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\pydantic_settings\sources\providers\env.py", line 111, in prepare_field_value
env_val_built = self.explode_env_vars(field_name, field, self.env_vars)
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\pydantic_settings\sources\providers\env.py", line 256, in explode_env_vars
raise e
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\pydantic_settings\sources\providers\env.py", line 253, in explode_env_vars
env_val = self.decode_complex_value(last_key, target_field, env_val) # type: ignore
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\pydantic_settings\sources\base.py", line 186, in decode_complex_value
return json.loads(value)
~~~~~~~~~~^^^^^^^
File "C:\python_3.13.0\Lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
~~~~~~~~~~~~~~~~~~~~~~~^^^
File "C:\python_3.13.0\Lib\json\decoder.py", line 344, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "C:\python_3.13.0\Lib\json\decoder.py", line 360, in raw_decode
obj, end = self.scan_once(s, idx)
~~~~~~~~~~~~~~^^^^^^^^
json.decoder.JSONDecodeError: Invalid \escape: line 1 column 17 (char 16)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "c:\users\lautaro\.local\bin\ktoolbox.exe\__main__.py", line 2, in <module>
from ktoolbox.__main__ import main
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\ktoolbox\__main__.py", line 4, in <module>
from ktoolbox.cli import KToolBoxCli
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\ktoolbox\cli.py", line 12, in <module>
from ktoolbox.action import create_job_from_post, create_job_from_creator, generate_post_path_name
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\ktoolbox\action\__init__.py", line 1, in <module>
from .base import *
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\ktoolbox\action\base.py", line 3, in <module>
from ktoolbox.utils import BaseRet
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\ktoolbox\utils.py", line 15, in <module>
from ktoolbox.configuration import config
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\ktoolbox\configuration.py", line 305, in <module>
config = Configuration()
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\pydantic_settings\main.py", line 189, in __init__
**__pydantic_self__._settings_build_values(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
values,
^^^^^^^
...<25 lines>...
_secrets_dir=_secrets_dir,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\pydantic_settings\main.py", line 424, in _settings_build_values
source_state = source()
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\pydantic_settings\sources\providers\dotenv.py", line 106, in __call__
data: dict[str, Any] = super().__call__()
~~~~~~~~~~~~~~~~^^
File "C:\Users\DAN\pipx\venvs\ktoolbox\Lib\site-packages\pydantic_settings\sources\base.py", line 492, in __call__
raise SettingsError(
f'error parsing value for field "{field_name}" from source "{self.__class__.__name__}"'
) from e
pydantic_settings.exceptions.SettingsError: error parsing value for field "job" from source "DotEnvSettingsSource"POSSIBLE ENHANCEMENT:
Continuing on bug 2, I want to add to "KTOOLBOX_JOB__EXTERNAL_LINK_PATTERNS=" the next website:
"https?://catbox\.moe/[^\\s]+",
https://catbox.moe/
But instead of adding websites as a whole from a list, why not detect if "http" is inside of the provided text in a post. This was how I used to detect all links inside of posts, extract all contents in the forms of a .json, transform all .json to .txt, advanced search for term "http" trough windows, copy all results into a .txt file.
Not too sure how easy or hard this would be in the current iteration of KToolBox, but, I leave this idea here just in case.