Minor adjsutments to cursor rules (#630)

* Minor adjsutments

* fix typing

---------

Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>

Author: anuunchin

ai/cursor/.cursor/rules/build-rest-api.mdc

@@ -1,31 +1,32 @@
 ---
-description: 
+description: Crucial guidelines to build a dlt rest api source
 globs: 
 alwaysApply: true
 ---
 ## Prerequisities to writing a source
 
-1. VERY IMPORTANT. When writing new source, you should have example available in rest_api_pipeline.py file. Use github rest api source for the general structure of the code. If you do not see this file, ask user to add it
-2. Recall OpenAPI spec. You will need to figure out the same information that OpenAPI spec contains on each API.
+1. VERY IMPORTANT. When writing a new source, you should have an example available in the rest_api_pipeline.py file. 
+Use this example or the github rest api source example from dlt's documentation on rest api for the general structure of the code. If you do not see this file rest_api_pipeline.py, ask the user to add it
+2. Recall OpenAPI spec. You will figure out the same information that the OpenAPI spec contains for each API.
 3. In particular:
 - API base url
 - type of authentication
 - list of endpoints with method GET (you can read data for those)
-4. You will need to figure additional information that is required for successful data extraction
+4. You will figure out additional information that is required for successful data extraction
 - type of pagination
 - if data from an endpoint can be loaded incrementally
 - unwrapping end user data from a response
 - write disposition of the endpoint: append, replace, merge
 - in case of merge, you need to find primary key that can be compound
-5. Some endpoints take data from other endpoints. For example `comments` endpoint needs `post id` to get list of comments per particular post. You'll need to figure out such connections
-6. **ASK USER IF YOU MISS CRUCIAL INFORMATION** You should make sure user provided you with enough information to figure out the above. Below are the most common possibilities
+5. Some endpoints take data from other endpoints. For example, in the github rest api source example from dlt's documentation, the `comments` endpoint needs `post id` to get the list of comments per particular post. You'll need to figure out such connections
+6. **ASK USER IF YOU MISS CRUCIAL INFORMATION** You will make sure the user has provided you with enough information to figure out the above. Below are the most common possibilities
 - open api spec (file or link)
 - any other api definition, for example Airbyte low code yaml
 - a source code in Python, java or c# of such connector or API client
 - a documentation of the api or endpoint
 7. In case you find more than 10 endpoints and you do not get instructions which you should add to the source, ask user.
-8. please make sure you use right pagination and you use exactly the arguments that are available in pagination guide. do not try to guess anything. remember that we have many paginator types that are configured differently
-9. When creating pipeline instance add progress="log" as parameter
+8. Make sure you use the right pagination and use exactly the arguments that are available in the pagination guide. do not try to guess anything. remember that we have many paginator types that are configured differently
+9. When creating pipeline instance add progress="log" as parameter `pipeline = dlt.pipeline(..., progress="log")`
 10. When fixing a bug report focus only on a single cause. ie. incremental, pagination or authentication or wrong dict fields
 11. You should have references for paginator types, authenticator types and general reference for rest api in you context. **DO NOT GUESS. DO NOT INVENT CODE. YOU SHOULD HAVE DOCUMENTATION FOR EVERYTHING YOU NEED. IF NOT - ASK USER**
 

ai/cursor/.cursor/rules/dlt.mdc

@@ -1,5 +1,5 @@
 ---
-description: 
+description: Information about dlt
 globs: 
 alwaysApply: true
 ---

ai/cursor/.cursor/rules/rest_api_extract_parameters.mdc

@@ -1,7 +1,7 @@
 ---
 description: This rule helps identify and extract ALL necessary parameters from API documentation to build a dlt REST API source
 globs: 
-alwaysApply: false
+alwaysApply: true
 ---
 # REST API Parameter Extraction Guide
 

ai/cursor/.cursor/rules/rest_api_pagination.mdc

@@ -1,7 +1,7 @@
 ---
 description: Use this rule when writing REST API Source to configure right pagination type for an Endpoint
 globs: 
-alwaysApply: false
+alwaysApply: true
 ---
 
 # dlt REST API Pagination Configuration Guide

sources/hubspot/__init__.py

@@ -134,7 +134,7 @@ def crm_objects(
 
 
 def crm_object_history(
-    object_type: THubspotObjectType,
+    object_type: str,
     api_key: str,
     props: Optional[Sequence[str]] = None,
     include_custom_props: bool = True,
@@ -143,7 +143,7 @@ def crm_object_history(
     Fetch the history of property changes for a given CRM object type.
 
     Args:
-        object_type (THubspotObjectType): Type of HubSpot object (e.g., 'company', 'contact').
+        object_type (str): Type of HubSpot object (e.g., 'company', 'contact').
         api_key (str, optional): API key for HubSpot authentication.
         props (Optional[Sequence[str]], optional): List of properties to retrieve. Defaults to None.
         include_custom_props (bool, optional): Include custom properties in the result. Defaults to True.
@@ -356,7 +356,7 @@ def pipelines_for_objects(
             Iterator[DltResource]: dlt resources for pipelines and stages.
         """
 
-        def get_pipelines(object_type: THubspotObjectType) -> Iterator[TDataItems]:
+        def get_pipelines(object_type: str) -> Iterator[TDataItems]:
             yield from fetch_data(
                 CRM_PIPELINES_ENDPOINT.format(objectType=object_type),
                 api_key=api_key_inner,

sources/jira/__init__.py

@@ -33,7 +33,7 @@ def jira(
         res_function = dlt.resource(
             get_paginated_data, name=endpoint_name, write_disposition="replace"
         )(
-            **endpoint_parameters,
+            **endpoint_parameters,  # type: ignore[arg-type]
             subdomain=subdomain,
             email=email,
             api_token=api_token,

sources/pipedrive/__init__.py

@@ -94,7 +94,8 @@ def pipedrive_source(
         name="deals_flow", write_disposition="merge", primary_key="id"
     )(_get_deals_flow)(pipedrive_api_key)
 
-    yield leads(pipedrive_api_key, update_time=since_timestamp)
+    # if simple value is passed in place of incremental, it will be used as initial value
+    yield leads(pipedrive_api_key, update_time=since_timestamp)  # type: ignore[arg-type]
 
 
 def _get_deals_flow(

sources/unstructured_data/__init__.py

@@ -2,7 +2,7 @@
 
 import asyncio
 import os
-from typing import Dict
+from typing import Dict, Optional
 
 import dlt
 from dlt.common import logger
@@ -18,7 +18,7 @@
 
 @dlt.resource
 def unstructured_to_structured_resource(
-    queries: Dict[str, str] = INVOICE_QUERIES,
+    queries: Optional[Dict[str, str]] = dlt.config.value,
     openai_api_key: str = dlt.secrets.value,
     vectorstore: str = "chroma",
     table_name: str = "unstructured_to_structured_resource",
@@ -43,6 +43,8 @@ def unstructured_to_structured_resource(
     """
     if openai_api_key:
         os.environ["OPENAI_API_KEY"] = openai_api_key
+    if queries is None:
+        queries = dict(INVOICE_QUERIES)
 
     return dlt.transformer(
         convert_data,