Merge pull request #607 from ScrapeGraphAI/pre/beta

abstract graph fix

Author: f-aguzzi

CHANGELOG.md

@@ -1,3 +1,81 @@
+## [1.15.1-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0...v1.15.1-beta.1) (2024-08-28)
+
+
+### Bug Fixes
+
+* abstract graph local model ([04128e7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/04128e7e9f585aaf774fabf646c4d9d3b96b8333))
+* **models:** better DeepSeek and OneApi integration ([f7a85c2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f7a85c266ae758cc16297ebc5d98f8919a80c523))
+* **docloaders:** BrowserBase dynamic import ([5c16ee9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5c16ee985b11948c6a8c1dbfd051d458fa193973))
+* bug for abstract graph ([cf73883](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf73883451729b19034005ee7ebe618c1e256a11))
+* **AbstractGraph:** correct and simplify instancing logic ([f73343f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f73343f19386b31878706963597c2565a023068d))
+* **BurrBrige:** dynamic imports ([7789663](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7789663338a89d27fde322ae282ce07ccca16845))
+* **AbstractGraph:** model selection bug ([4f120e2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4f120e29c546373a2cc06c102cc9886cc5270c06))
+* set up dynamic imports correctly ([83e71df](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/83e71df2e2cb3b6bfba11f8879d5c4917a3e1837))
+
+
+### chore
+
+* **examples:** update model names ([f6df9b7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f6df9b75125b4cacbef4af29faf3e17a13ff108c))
+* update README.md ([5f562b8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5f562b89bd63eba1300afe98572f152a0621b370))
+
+
+### Test
+
+* **AbstractGraph:** add AbstractGraph tests ([229d74d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/229d74d4bd39befa3723fa2841e23d40007a9772))
+
+
+### CI
+
+* **release:** 1.15.0-beta.4 [skip ci] ([c1ce9c6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c1ce9c69d4ba746d488891d18fa64460e76124bf))
+* **release:** 1.15.0-beta.5 [skip ci] ([22ab45f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/22ab45f6bda3a12ab01c743fd124448a2e26cd46))
+* **release:** 1.15.0-beta.6 [skip ci] ([050fa3f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/050fa3faa02cb2a86ce7c0f61c99e4fa8cf3f9a5))
+* **release:** 1.15.0-beta.7 [skip ci] ([be3f1ec](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/be3f1ec58d6354d583401f51f310f6aac987a393))
+* **release:** 1.15.0-beta.8 [skip ci] ([dbec550](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/dbec55064feac8dfe01290bf82b5b47b013b589d))
+
+## [1.15.0-beta.8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.7...v1.15.0-beta.8) (2024-08-28)
+
+
+### Bug Fixes
+
+* **models:** better DeepSeek and OneApi integration ([f7a85c2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f7a85c266ae758cc16297ebc5d98f8919a80c523))
+* **AbstractGraph:** model selection bug ([4f120e2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4f120e29c546373a2cc06c102cc9886cc5270c06))
+
+## [1.15.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.6...v1.15.0-beta.7) (2024-08-27)
+
+
+### Bug Fixes
+
+* bug for abstract graph ([cf73883](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf73883451729b19034005ee7ebe618c1e256a11))
+
+## [1.15.0-beta.6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.5...v1.15.0-beta.6) (2024-08-27)
+
+
+### Bug Fixes
+
+* **docloaders:** BrowserBase dynamic import ([5c16ee9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5c16ee985b11948c6a8c1dbfd051d458fa193973))
+* **AbstractGraph:** correct and simplify instancing logic ([f73343f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f73343f19386b31878706963597c2565a023068d))
+* **BurrBrige:** dynamic imports ([7789663](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7789663338a89d27fde322ae282ce07ccca16845))
+* set up dynamic imports correctly ([83e71df](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/83e71df2e2cb3b6bfba11f8879d5c4917a3e1837))
+
+
+### chore
+
+* **examples:** update model names ([f6df9b7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f6df9b75125b4cacbef4af29faf3e17a13ff108c))
+
+
+### Test
+
+* **AbstractGraph:** add AbstractGraph tests ([229d74d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/229d74d4bd39befa3723fa2841e23d40007a9772))
+
+## [1.15.0-beta.5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.4...v1.15.0-beta.5) (2024-08-26)
+
+
+### Bug Fixes
+
+* abstract graph local model ([04128e7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/04128e7e9f585aaf774fabf646c4d9d3b96b8333))
+
+## [1.15.0-beta.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.3...v1.15.0-beta.4) (2024-08-26)
+
 ## [1.15.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.1...v1.15.0) (2024-08-26)
 
 
@@ -21,47 +99,39 @@
 * **release:** 1.15.0-beta.2 [skip ci] ([ab21576](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ab215764353773c5303b88743c6cca4fa7e1b52e))
 * **release:** 1.15.0-beta.3 [skip ci] ([132ee5b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/132ee5b7daf36ef376bfbc63bc6dc7f2332fdd6b))
 
-## [1.15.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.2...v1.15.0-beta.3) (2024-08-24)
-
-
 
 ### Bug Fixes
 
-* update abstract graph ([86fe5fc](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/86fe5fcaf1a6ba28786678874378f07fba1db40f))
-
-## [1.15.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.1...v1.15.0-beta.2) (2024-08-23)
-
+* add claude3.5 sonnet ([ee8f8b3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ee8f8b31ecfe4ffd311528d2f48cb055e4609d99))
 
-### Bug Fixes
 
-* abstract graph ([cf1fada](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf1fada36a6716cb0e24bbc5da7509446a964145))
+### CI
 
+* **release:** 1.14.1 [skip ci] ([88e76ce](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/88e76ceedb39dc1b41222e9a5cb8a6f0d81cadf4))
 
+## [1.14.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0...v1.14.1) (2024-08-24)
 
-### Docs
 
-* added sponsors ([b3a2d0d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b3a2d0d65a41f6e645fac3fc84f702fdf64b951c))
 
-## [1.15.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.1-beta.1...v1.15.0-beta.1) (2024-08-23)
+### Bug Fixes
 
 
-### Features
+* update abstract graph ([86fe5fc](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/86fe5fcaf1a6ba28786678874378f07fba1db40f))
 
-* ligthweigthing the library ([62f32e9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/62f32e994bcb748dfef4f7e1b2e5213a989c33cc))
+## [1.15.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0-beta.1...v1.15.0-beta.2) (2024-08-23)
 
 
 ### Bug Fixes
 
-* Azure OpenAI issue ([a92b9c6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/a92b9c6970049a4ba9dbdf8eff3eeb7f98c6c639))
-
-## [1.14.1-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0...v1.14.1-beta.1) (2024-08-21)
+* abstract graph ([cf1fada](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf1fada36a6716cb0e24bbc5da7509446a964145))
 
 
-### Bug Fixes
+### Docs
 
-* **models_tokens:** add llama2 and llama3 sizes explicitly ([b05ec16](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b05ec16b252d00c9c9ee7c6d4605b420851c7754))
+* added sponsors ([b3a2d0d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b3a2d0d65a41f6e645fac3fc84f702fdf64b951c))
 
 
+#
 ## [1.14.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.3...v1.14.0) (2024-08-20)
 
 

examples/anthropic/csv_scraper_graph_multi_anthropic.py renamed to examples/anthropic/csv_scraper_graph_multi_haiku.py

@@ -27,7 +27,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000},
+    },
 }
 
 # ************************************************

examples/anthropic/csv_scraper_anthropic.py renamed to examples/anthropic/csv_scraper_haiku.py

@@ -33,8 +33,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ************************************************

examples/anthropic/custom_graph_anthropic.py renamed to examples/anthropic/custom_graph_haiku.py

@@ -5,10 +5,9 @@
 import os
 from dotenv import load_dotenv
 
-from langchain_openai import OpenAIEmbeddings
-from langchain_openai import ChatOpenAI
+from langchain_anthropic import ChatAnthropic
 from scrapegraphai.graphs import BaseGraph
-from scrapegraphai.nodes import FetchNode, ParseNode, RAGNode, GenerateAnswerNode, RobotsNode
+from scrapegraphai.nodes import FetchNode, ParseNode, GenerateAnswerNode, RobotsNode
 load_dotenv()
 
 # ************************************************
@@ -18,17 +17,15 @@
 graph_config = {
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
-        "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+        "model": "claude-3-haiku-20240307",
+    },
 }
 
 # ************************************************
 # Define the graph nodes
 # ************************************************
 
-llm_model = OpenAI(graph_config["llm"])
-embedder = OpenAIEmbeddings(api_key=llm_model.openai_api_key)
+llm_model = ChatAnthropic(graph_config["llm"])
 
 # define the nodes for the graph
 robot_node = RobotsNode(
@@ -57,15 +54,6 @@
         "verbose": True,
     }
 )
-rag_node = RAGNode(
-    input="user_prompt & (parsed_doc | doc)",
-    output=["relevant_chunks"],
-    node_config={
-        "llm_model": llm_model,
-        "embedder_model": embedder,
-        "verbose": True,
-    }
-)
 generate_answer_node = GenerateAnswerNode(
     input="user_prompt & (relevant_chunks | parsed_doc | doc)",
     output=["answer"],
@@ -84,14 +72,12 @@
         robot_node,
         fetch_node,
         parse_node,
-        rag_node,
         generate_answer_node,
     ],
     edges=[
         (robot_node, fetch_node),
         (fetch_node, parse_node),
-        (parse_node, rag_node),
-        (rag_node, generate_answer_node)
+        (parse_node, generate_answer_node)
     ],
     entry_point=robot_node
 )

examples/anthropic/json_scraper_anthropic.py renamed to examples/anthropic/json_scraper_haiku.py

@@ -27,8 +27,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ************************************************

examples/anthropic/json_scraper_multi_anthropic.py renamed to examples/anthropic/json_scraper_multi_haiku.py

@@ -12,8 +12,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 FILE_NAME = "inputs/example.json"

examples/anthropic/pdf_scraper_graph_anthropic.py renamed to examples/anthropic/pdf_scraper_graph_haiku.py

@@ -15,8 +15,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 source = """

examples/anthropic/pdf_scraper_multi_anthropic.py renamed to examples/anthropic/pdf_scraper_multi_haiku.py

@@ -12,8 +12,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ***************

examples/anthropic/scrape_plain_text_anthropic.py renamed to examples/anthropic/scrape_plain_text_haiku.py

@@ -29,8 +29,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ************************************************

examples/anthropic/script_generator_anthropic.py renamed to examples/anthropic/script_generator_haiku.py

@@ -17,8 +17,7 @@
     "llm": {
         "api_key": os.getenv("ANTHROPIC_API_KEY"),
         "model": "anthropic/claude-3-haiku-20240307",
-        "max_tokens": 4000
-        },
+    },
 }
 
 # ************************************************