LLM-REVal: Can We Trust LLM Reviewers Yet?

▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
█  ╭─────────────────────────────────────────────────────╮  █
█  ⚡        Imaginary Conference of Lost Reality         ⚡ █
█  ╰─────────────────────────────────────────────────────╯  █
█     _     _     __  __    ____  _______     __    _        █
█    | |   | |   |  \/  |  |  _ \| ____\ \   / /_ _| |      █
█    | |   | |   | |\/| |  | |_) |  _|  \ \ / / _  | |      █
█    | |___| |___| |  | |  |  _ <| |___  \ V / (_| | |___   █
█    |_____|_____|_|  |_|  |_| \_\_____|  \_/ \____|\____|   █
█  ╭─────────────────────────────────────────────────────╮  █
█  🚀             LLM REViewer Re-EValuation             🚀 █
█  ╰─────────────────────────────────────────────────────╯  █
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

-----

0. Features Included in This Repository:

• Idea generation
• Paper generation
• Paper review
• Paper revision

cd LLM-REVal

---

1. Local LaTeX Environment Setup

Please install a local LaTeX environment, such as TeX Live, which is necessary for compiling the generated papers.

Reference Link: TeX Live Official Website

---

2. Research Agent Deployment

2.1. LLM Idea Generation

For more granular control, refer to the idea_generation.sh file, which includes adjustable parameters.

bash idea_generation.sh

(Implementation based on AI-Researcher)

Example Idea:

An example of an idea can be found here: idea_example

2.2. LLM-Authored Paper Generation

For more granular control, refer to the paper_generation.sh file, which includes adjustable parameters.

bash paper_generation.sh

(Some prompts reference AI-Scientist)

An example of a generated paper can be found here: paper_generation_example

2.3. Paper Revision

For more granular control, refer to the paper_revise.sh file, which includes adjustable parameters.

Run this script to revise a paper based on a set of reviews:

bash paper_revise.sh

---

3. Review Agent Deployment

For more granular control, refer to the paper_review.sh file, which includes adjustable parameters.

bash paper_review.sh

(Implementation based on AgentReview)

An example of reviews can be found here: paper_review_example