Den 450 move llm logic to sdk

README.md

@@ -4,12 +4,7 @@
   <a href="https://discord.gg/ETPBdXU3kx"><img src="https://img.shields.io/badge/Discord-Join%20Us-7289DA?style=for-the-badge&logo=discord&logoColor=white" alt="Discord"></a>
 </p>
 
-<div align="center">
-<h2>🎉We are going open source🎉</h1>
-<p>
-   Let us know if you're interested in contributing! We're working on integrating the core logic for getting elements and extraction into the sdk!
-</p>
-</div>
+> **Notice:** The Dendrite SDK is not under active development anymore. However, the project will remain fully open source so that you and others can learn from it. Feel free to fork, study, or adapt this code for your own projects as you wish – reach out to us on Discord if you have questions! We love chatting about web AI agents. 🤖
 
 ## What is Dendrite?
 
@@ -24,33 +19,60 @@
 
 #### A simple outlook integration
 
-With Dendrite, it's easy to create web interaction tools for your agent.
+With Dendrite it's easy to create web interaction tools for your agent.
+
+Here's how you can send an email:
 
 ```python
-from dendrite import Dendrite
+from dendrite import AsyncDendrite
+
 
-def send_email():
-    client = Dendrite(auth="outlook.live.com")
+async def send_email(to, subject, message):
+    client = AsyncDendrite(auth="outlook.live.com")
 
     # Navigate
-    client.goto(
-        "https://outlook.live.com/mail/0/",
-        expected_page="An email inbox"
+    await client.goto(
+        "https://outlook.live.com/mail/0/", expected_page="An email inbox"
     )
 
     # Create new email and populate fields
-    client.click("The new email button")
-    client.fill_fields({
-        "Recipient": to,
-        "Subject": subject,
-        "Message": message
-    })
+    await client.click("The new email button")
+    await client.fill("The recipient field", to)
+    await client.press("Enter")
+    await client.fill("The subject field", subject)
+    await client.fill("The message field", message)
 
     # Send email
-    client.click("The send button")
+    await client.press("Enter", hold_cmd=True)
+
+
+if __name__ == "__main__":
+    import asyncio
+
+    asyncio.run(send_email("test@example.com", "Hello", "This is a test email"))
+
 ```
 
-To authenticate you'll need to use our Chrome Extension **Dendrite Vault**, you can download it [here](https://chromewebstore.google.com/detail/dendrite-vault/faflkoombjlhkgieldilpijjnblgabnn). Read more about authentication [in our docs](https://docs.dendrite.systems/examples/authentication-instagram).
+You'll need to add your own Anthropic key or [configure which LLMs to use yourself](https://docs.dendrite.systems/concepts/config).
+
+
+```.env
+ANTHROPIC_API_KEY=sk-...
+```
+
+To **authenticate** on any web service with Dendrite, follow these steps:
+
+1. Run the authentication command
+
+  ```bash
+  dendrite auth --url outlook.live.com
+  ```
+
+2. This command will open a browser that you'll be able to login with.
+
+3. After you've logged in, press enter in your terminal. This will save your cookies locally so that they can be used in your code.
+
+Read more about authentication [in our docs](https://docs.dendrite.systems/examples/authentication).
 
 ## Quickstart
 
@@ -60,81 +82,159 @@ pip install dendrite && dendrite install
 
 #### Simple navigation and interaction
 
-Initialize the Dendrite client and start doing web interactions without boilerplate.
-
-[Get your API key here](https://dendrite.systems/app)
-
 ```python
-from dendrite import Dendrite
+from dendrite import AsyncDendrite
+
+async def main():
+    client = AsyncDendrite()
 
-client = Dendrite(dendrite_api_key="sk...")
+    await client.goto("https://google.com")
+    await client.fill("Search field", "Hello world")
+    await client.press("Enter")
 
-client.goto("https://google.com")
-client.fill("Search field", "Hello world")
-client.press("Enter")
+if __name__ == "__main__":
+    import asyncio
+    asyncio.run(main())
 ```
 
 In the example above, we simply go to Google, populate the search field with "Hello world" and simulate a keypress on Enter. It's a simple example that starts to explore the endless possibilities with Dendrite. Now you can create tools for your agents that have access to the full web without depending on APIs.
 
-## More powerful examples
+## More Examples
 
-Now, let's have some fun. Earlier we showed you a simple send_email example. And sending emails is cool, but if that's all our agent can do it kind of sucks. So let's create two cooler examples.
+### Get any page as markdown
 
-### Download Bank Transactions
+This is a simple example of how to get any page as markdown, great for feeding to an LLM.
+
+```python
+from dendrite import AsyncDendrite
+from dotenv import load_dotenv
+
+async def main():
+    browser = AsyncDendrite()
+
+    await browser.goto("https://dendrite.systems")
+    await browser.wait_for("the page to load")
+
+    # Get the entire page as markdown
+    md = await browser.markdown()
+    print(md)
+    print("=" * 200)
+
+    # Only get a certain part of the page as markdown
+    data_extraction_md = await browser.markdown("the part about data extraction")
+    print(data_extraction_md)
+
+if __name__ == "__main__":
+    import asyncio
+    asyncio.run(main())
+```
 
-First up, a tool that allows our AI agent to download our bank's monthly transactions so that they can be analyzed and compiled into a report that can be sent to stakeholders with `send_email`.
+### Get Company Data from Y Combinator
+
+The classic web data extraction test, made easy:
 
 ```python
-from dendrite import Dendrite
+from dendrite import AsyncDendrite
+import pprint
+import asyncio
 
-def get_transactions() -> str:
-    client = Dendrite(auth="mercury.com")
 
-    # Navigate and wait for loading
-    client.goto(
-      "https://app.mercury.com/transactions",
-      expected_page="Dashboard with transactions"
-    )
-    client.wait_for("The transactions to finish loading")
+async def main():
+    browser = AsyncDendrite()
 
-    # Modify filters
-    client.click("The 'add filter' button")
-    client.click("The 'show transactions for' dropdown")
-    client.click("The 'this month' option")
+    # Navigate
+    await browser.goto("https://www.ycombinator.com/companies")
+
+    # Find and fill the search field with "AI agent"
+    await browser.fill(
+        "Search field", value="AI agent"
+    )  # Element selector cached since before
+    await browser.press("Enter")
+
+    # Extract startups with natural language description
+    # Once created by our agent, the same script will be cached and reused
+    startups = await browser.extract(
+        "All companies. Return a list of dicts with name, location, description and url"
+    )
+    pprint.pprint(startups, indent=2)
 
-    # Download file
-    client.click("The 'export filtered' button")
-    transactions = client.get_download()
 
-    # Save file locally
-    path = "files/transactions.xlsx"
-    transactions.save_as(path)
+if __name__ == "__main__":
+    asyncio.run(main())
 
-    return path
+```
 
-def analyze_transactions(path: str):
-    ...
+returns 
 ```
+[ { 'description': 'Book accommodations around the world.',
+    'location': 'San Francisco, CA, USA',
+    'name': 'Airbnb',
+    'url': 'https://www.ycombinator.com/companies/airbnb'},
+  { 'description': 'Digital Analytics Platform',
+    'location': 'San Francisco, CA, USA',
+    'name': 'Amplitude',
+    'url': 'https://www.ycombinator.com/companies/amplitude'},
+...
+] }
+```
+
 
-### Extract Google Analytics
+### Extract Data from Google Analytics
 
-Finally, it would be cool if we could add the amount of monthly visitors from Google Analytics to our report. We can do that by using the `extract` function:
+Here's how to get the amount of monthly visitors from Google Analytics using the `extract` function:
 
 ```python
-def get_visitor_count() -> int:
-    client = Dendrite(auth="analytics.google.com")
+async def get_visitor_count() -> int:
+    client = AsyncDendrite(auth="analytics.google.com")
 
-    client.goto(
+    await client.goto(
       "https://analytics.google.com/analytics/web",
       expected_page="Google Analytics dashboard"
     )
 
     # The Dendrite extract agent will create a web script that is cached
     # and reused. It will self-heal when the website updates
-    visitor_count = client.extract("The amount of visitors this month", int)
+    visitor_count = await client.extract("The amount of visitors this month", int)
     return visitor_count
 ```
 
+### Download Bank Transactions
+
+Here's tool that allows our AI agent to download our bank's monthly transactions so that they can be analyzed and compiled into a report.
+
+```python
+from dendrite import AsyncDendrite
+
+async def get_transactions() -> str:
+    client = AsyncDendrite(auth="mercury.com")
+
+    # Navigate and wait for loading
+    await client.goto(
+      "https://app.mercury.com/transactions",
+      expected_page="Dashboard with transactions"
+    )
+    await client.wait_for("The transactions to finish loading")
+
+    # Modify filters
+    await client.click("The 'add filter' button")
+    await client.click("The 'show transactions for' dropdown")
+    await client.click("The 'this month' option")
+
+    # Download file
+    await client.click("The 'export filtered' button")
+    transactions = await client.get_download()
+
+    # Save file locally
+    path = "files/transactions.xlsx"
+    await transactions.save_as(path)
+
+    return path
+
+async def analyze_transactions(path: str):
+    ... # Analyze the transactions with LLM of our choice
+```
+
+
 ## Documentation
 
 [Read the full docs here](https://docs.dendrite.systems)
@@ -145,7 +245,7 @@ def get_visitor_count() -> int:
 
 When you want to scale up your AI agents, we support using browsers hosted by Browserbase. This way you can run many agents in parallel without having to worry about the infrastructure.
 
-To start using Browserbase just swap out the `Dendrite` class with `DendriteRemoteBrowser` and add your Browserbase API key and project id, either in the code or in a `.env` file like this:
+To start using Browserbase just swap out the `AsyncDendrite` class with `AsyncDendriteRemoteBrowser` and add your Browserbase API key and project id, either in the code or in a `.env` file like this:
 
 ```bash
 # ... previous keys
@@ -154,17 +254,19 @@ BROWSERBASE_PROJECT_ID=
 ```
 
 ```python
-# from dendrite import Dendrite
-from dendrite import DendriteRemoteBrowser
-
-...
-
-# client = Dendrite(...)
-client = DendriteRemoteBrowser(
-    # Use interchangeably with the Dendrite class
-    browserbase_api_key="...", # or specify the browsebase keys in the .env file
-    browserbase_project_id="..."
-)
+# from dendrite import AsyncDendrite
+from dendrite import AsyncDendriteRemoteBrowser
+
+async def main():
+    # client = AsyncDendrite(...)
+    client = AsyncDendriteRemoteBrowser(
+        # Use interchangeably with the AsyncDendrite class
+        browserbase_api_key="...", # or specify the browsebase keys in the .env file
+        browserbase_project_id="..."
+    )
+    ...
 
-...
+if __name__ == "__main__":
+    import asyncio
+    asyncio.run(main())
 ```

dendrite/__init__.py

@@ -1,33 +1,22 @@
 import sys
-from loguru import logger
-from dendrite.async_api import (
-    AsyncDendrite,
-    AsyncElement,
-    AsyncPage,
-    AsyncElementsResponse,
-)
 
-from dendrite.sync_api import (
+from dendrite._loggers.d_logger import logger
+from dendrite.browser.async_api import AsyncDendrite, AsyncElement, AsyncPage
+from dendrite.logic.config import Config
+
+from dendrite.browser.sync_api import (
     Dendrite,
     Element,
     Page,
-    ElementsResponse,
 )
 
-logger.remove()
-
-fmt = "<green>{time: HH:mm:ss.SSS}</green> | <level>{level: <8}</level>- <level>{message}</level>"
-
-logger.add(sys.stderr, level="INFO", format=fmt)
-
 
 __all__ = [
     "AsyncDendrite",
     "AsyncElement",
     "AsyncPage",
-    "AsyncElementsResponse",
     "Dendrite",
     "Element",
     "Page",
-    "ElementsResponse",
+    "Config",
 ]