Skip to content

chore: Migrate to ty type checker #1625

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
vdusek wants to merge 5 commits into
base: master
Choose a base branch
from
Open

chore: Migrate to ty type checker #1625

vdusek wants to merge 5 commits into from
+344 −386

Conversation

@vdusek
Copy link
Collaborator

@vdusek vdusek commented Dec 17, 2025

No description provided.

@vdusek vdusek added this to the 129th sprint - Tooling team milestone Dec 17, 2025
@vdusek vdusek requested review from Mantisus and janbuchar December 17, 2025 16:26
@vdusek vdusek self-assigned this Dec 17, 2025
@vdusek vdusek added t-tooling Issues with this label are in the ownership of the tooling team. adhoc Ad-hoc unplanned task added during the sprint. labels Dec 17, 2025
@codecov
Copy link

codecov bot commented Dec 17, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 85.82090% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.41%. Comparing base (6dbfcb5) to head (71355ea).

Files with missing lines Patch % Lines
src/crawlee/http_clients/_curl_impersonate.py 57.57% 14 Missing ⚠️
.../crawlee/storage_clients/_redis/_storage_client.py 66.66% 3 Missing ⚠️
.../crawlee/storage_clients/_redis/_dataset_client.py 66.66% 1 Missing ⚠️
...ee/storage_clients/_redis/_request_queue_client.py 66.66% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #1625      +/-   ##
==========================================
- Coverage   92.49%   92.41%   -0.08%     
==========================================
  Files         157      157              
  Lines       10440    10478      +38     
==========================================
+ Hits         9656     9683      +27     
- Misses        784      795      +11
Flag Coverage Δ
unit 92.41% <85.82%> (-0.08%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Mantisus
Mantisus approved these changes Dec 17, 2025
View reviewed changes
Copy link
Collaborator

@Mantisus Mantisus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job!

@vdusek vdusek changed the title chore: Migrate to ty type checker chore: Migrate to ty type checker Dec 19, 2025
@vdusek vdusek requested a review from Pijukatel January 5, 2026 08:03
janbuchar
janbuchar approved these changes Jan 5, 2026
View reviewed changes
Copy link
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of nits but probably OK. Will ty let us know when a type-ignore comment becomes obsolete?

Pijukatel
Pijukatel requested changes Jan 5, 2026
View reviewed changes
Copy link
Collaborator

@Pijukatel Pijukatel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to solve some ignores. I added bunch of suggestions and raised some issues in ty repo

@vdusek
Copy link
Collaborator Author

vdusek commented Jan 7, 2026

Will ty let us know when a type-ignore comment becomes obsolete?

Not now, see astral-sh/ty#278 and astral-sh/ty#1501 for more context.

@vdusek vdusek requested a review from Pijukatel January 7, 2026 14:14
@janbuchar
Copy link
Collaborator

janbuchar commented Jan 7, 2026

Will ty let us know when a type-ignore comment becomes obsolete?

Not now, see astral-sh/ty#278 and astral-sh/ty#1501 for more context.

Can we make an issue to revisit the ignores in ~2 months or something then?

@vdusek vdusek mentioned this pull request Jan 7, 2026
Open
@vdusek
Copy link
Collaborator Author

vdusek commented Jan 7, 2026

Will ty let us know when a type-ignore comment becomes obsolete?

Not now, see astral-sh/ty#278 and astral-sh/ty#1501 for more context.

Can we make an issue to revisit the ignores in ~2 months or something then?

Sure, -> #1664.

Mantisus
Mantisus reviewed Jan 7, 2026
View reviewed changes
src/crawlee/_utils/globs.py
seps = (os.path.sep, os.path.altsep) if os.path.altsep else os.path.sep

escaped_seps = ''.join(map(re.escape, seps))
escaped_seps = ''.join(map(re.escape, seps)) # ty: ignore[invalid-argument-type]
Copy link
Collaborator

@Mantisus Mantisus Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be resolved

_seps = ((os.path.sep, os.path.altsep) if os.path.altsep else (os.path.sep,)) if seps is None else seps

escaped_seps = ''.join(map(re.escape, _seps))

src/crawlee/crawlers/_abstract_http/_abstract_http_crawler.py
class _ParsedHttpCrawler(
AbstractHttpCrawler[ParsedHttpCrawlingContext[TParseResult], TParseResult, TSelectResult]
):
): # ty: ignore[invalid-generic-class]
Copy link
Collaborator

@Mantisus Mantisus Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

class _ParsedHttpCrawler(AbstractHttpCrawler):

Work here

src/crawlee/sessions/_cookies.py
if isinstance(cookies, dict):
for key, value in cookies.items():
self.set(key, value)
self.set(key, value) # ty: ignore[invalid-argument-type]
Copy link
Collaborator

@Mantisus Mantisus Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be resolved

self._jar = CookieJar()

if isinstance(cookies, list):
    for item in cookies:
        self.set(**item)

elif isinstance(cookies, SessionCookies):
    for cookie in cookies.jar:
        self._jar.set_cookie(cookie)

elif isinstance(cookies, dict):
    cookies_dict: dict[str, str] = cookies
    for key, value in cookies_dict.items():
        self.set(key, value)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@janbuchar janbuchar janbuchar approved these changes

@Pijukatel Pijukatel Awaiting requested review from Pijukatel

+1 more reviewer

@Mantisus Mantisus Mantisus approved these changes

Reviewers whose approvals may not affect merge requirements

Requested changes must be addressed to merge this pull request.

Assignees

@vdusek vdusek

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Milestone

129th sprint - Tooling team

Development

Successfully merging this pull request may close these issues.

5 participants

@vdusek @janbuchar @Mantisus @Pijukatel