Skip to content

Commit 0d028e2

Browse files
lwaekfjlklwaekfjlk
authored
feat(src): init class structure (#7)
* init class structure * init class structure * update
1 parent eeda774 commit 0d028e2

14 files changed

+2565
-5
lines changed

‎configs/coder_prompt.yaml

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
experiment_prompt: |
2+
Your goal is to implement the following idea: {title}.
3+
The proposed experiment is as follows: {idea}.
4+
You are given a total of up to {max_runs} runs to complete the necessary experiments. You do not need to use all {max_runs}.
5+
6+
First, plan the list of experiments you would like to run. For example, if you are sweeping over a specific hyperparameter, plan each value you would like to test for each run.
7+
8+
Note that we already provide the vanilla baseline results, so you do not need to re-run it.
9+
10+
For reference, the baseline results are as follows:
11+
12+
{baseline_results}
13+
14+
After you complete each change, we will run the command `python experiment.py --out_dir=run_i' where i is the run number and evaluate the results.
15+
YOUR PROPOSED CHANGE MUST USE THIS COMMAND FORMAT, DO NOT ADD ADDITIONAL COMMAND LINE ARGS.
16+
You can then implement the next thing on your list.
17+
18+
experiment_success_prompt: |
19+
Run {run_num} completed. Here are the results:
20+
{results}
21+
22+
Decide if you need to re-plan your experiments given the result (you often will not need to).
23+
24+
Someone else will be using `notes.txt` to perform a writeup on this in the future.
25+
Please include *all* relevant information for the writeup on Run {run_num}, including an experiment description and the run number. Be as verbose as necessary.
26+
27+
Then, implement the next thing on your list.
28+
We will then run the command `python experiment.py --out_dir=run_{next_run}'.
29+
YOUR PROPOSED CHANGE MUST USE THIS COMMAND FORMAT, DO NOT ADD ADDITIONAL COMMAND LINE ARGS.
30+
If you are finished with experiments, respond with 'ALL_COMPLETED'.
31+
32+
experiment_error_prompt: |
33+
Run failed with the following error {error}
34+
35+
experiment_timeout_prompt: |
36+
Run timed out after {timeout} seconds
37+
38+
plot_initial_prompt: |
39+
Great job! Please modify `plot.py` to generate the most relevant plots for the final writeup.
40+
41+
In particular, be sure to fill in the "labels" dictionary with the correct names for each run that you want to plot.
42+
43+
Only the runs in the `labels` dictionary will be plotted, so make sure to include all relevant runs.
44+
45+
We will be running the command `python plot.py` to generate the plots.
46+
47+
plot_error_prompt: |
48+
Plotting failed with the following error {error}
49+
50+
plot_timeout_prompt: |
51+
Plotting timed out after {timeout} seconds
52+
53+
notes_prompt: |
54+
Please modify `notes.txt` with a description of what each plot shows along with the filename of the figure. Please do so in-depth.
55+
56+
Somebody else will be using `notes.txt` to write a report on this in the future.

‎configs/reviewer_prompt.yaml

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
reviewer_system_prompt_base: >
2+
You are an AI researcher who is reviewing a paper that was submitted to a prestigious ML venue. Be critical and cautious in your decision.
3+
reviewer_system_prompt_neg: >
4+
You are an AI researcher who is reviewing a paper that was submitted to a prestigious ML venue. Be critical and cautious in your decision. If a paper is bad or you are unsure, give it bad scores and reject it.
5+
reviewer_system_prompt_pos: >
6+
You are an AI researcher who is reviewing a paper that was submitted to a prestigious ML venue. Be critical and cautious in your decision. If a paper is good or you are unsure, give it good scores and accept it.
7+
template_instructions: |
8+
Respond in the following format:
9+
10+
THOUGHT:
11+
<THOUGHT>
12+
13+
REVIEW JSON:
14+
```json
15+
<JSON>
16+
```
17+
18+
In <THOUGHT>, first briefly discuss your intuitions and reasoning for the evaluation.
19+
Detail your high-level arguments, necessary choices and desired outcomes of the review.
20+
Do not make generic comments here, but be specific to your current paper.
21+
Treat this as the note-taking phase of your review.
22+
23+
In <JSON>, provide the review in JSON format with the following fields in the order:
24+
- "Summary": A summary of the paper content and its contributions.
25+
- "Strengths": A list of strengths of the paper.
26+
- "Weaknesses": A list of weaknesses of the paper.
27+
- "Originality": A rating from 1 to 4 (low, medium, high, very high).
28+
- "Quality": A rating from 1 to 4 (low, medium, high, very high).
29+
- "Clarity": A rating from 1 to 4 (low, medium, high, very high).
30+
- "Significance": A rating from 1 to 4 (low, medium, high, very high).
31+
- "Questions": A set of clarifying questions to be answered by the paper authors.
32+
- "Limitations": A set of limitations and potential negative societal impacts of the work.
33+
- "Ethical Concerns": A boolean value indicating whether there are ethical concerns.
34+
- "Soundness": A rating from 1 to 4 (poor, fair, good, excellent).
35+
- "Presentation": A rating from 1 to 4 (poor, fair, good, excellent).
36+
- "Contribution": A rating from 1 to 4 (poor, fair, good, excellent).
37+
- "Overall": A rating from 1 to 10 (very strong reject to award quality).
38+
- "Confidence": A rating from 1 to 5 (low, medium, high, very high, absolute).
39+
- "Decision": A decision that has to be one of the following: Accept, Reject.
40+
neurips_form: |
41+
## Review Form
42+
Below is a description of the questions you will be asked on the review form for each paper and some guidelines on what to consider when answering these questions.
43+
When writing your review, please keep in mind that after decisions have been made, reviews and meta-reviews of accepted papers and opted-in rejected papers will be made public.
44+
45+
1. Summary: Briefly summarize the paper and its contributions.
46+
2. Strengths and Weaknesses: Provide a thorough assessment of the paper's strengths and weaknesses.
47+
3. Originality: Rate from 1 to 4.
48+
4. Quality: Rate from 1 to 4.
49+
5. Clarity: Rate from 1 to 4.
50+
6. Significance: Rate from 1 to 4.
51+
7. Questions: List any clarifying questions.
52+
8. Limitations: List any limitations or potential negative societal impacts.
53+
9. Ethical Concerns: Indicate whether there are ethical concerns.
54+
10. Soundness: Rate from 1 to 4.
55+
11. Presentation: Rate from 1 to 4.
56+
12. Contribution: Rate from 1 to 4.
57+
13. Overall: Rate from 1 to 10.
58+
14. Confidence: Rate from 1 to 5.
59+
15. Decision: Accept or Reject.
60+
61+
{{ template_instructions }}
62+
63+
meta_reviewer_system_prompt: |
64+
You are an Area Chair at a machine learning conference.
65+
You are in charge of meta-reviewing a paper that was reviewed by {reviewer_count} reviewers.
66+
Your job is to aggregate the reviews into a single meta-review in the same format.
67+
Be critical and cautious in your decision, find consensus, and respect the opinion of all the reviewers.
68+
69+
reviewer_reflection_prompt: |
70+
Round {current_round}/{num_reflections}.
71+
In your thoughts, first carefully consider the accuracy and soundness of the review you just created.
72+
Include any other factors that you think are important in evaluating the paper.
73+
Ensure the review is clear and concise, and the JSON is in the correct format.
74+
Do not make things overly complicated.
75+
In the next attempt, try and refine and improve your review.
76+
Stick to the spirit of the original review unless there are glaring issues.
77+
78+
Respond in the same format as before:
79+
THOUGHT:
80+
<THOUGHT>
81+
82+
REVIEW JSON:
83+
```json
84+
<JSON>
85+
```
86+
87+
If there is nothing to improve, simply repeat the previous JSON EXACTLY after the thought and include "I am done" at the end of the thoughts but before the JSON.
88+
ONLY INCLUDE "I am done" IF YOU ARE MAKING NO MORE CHANGES.
89+
90+
improvement_prompt: |
91+
The following review has been created for your research paper:
92+
"""
93+
{review}
94+
"""
95+
Improve the text using the review.

‎configs/thinker_prompt.yaml

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
idea_system_prompt: >
2+
You are an ambitious AI PhD student who is looking to publish a paper that will contribute significantly to the field.
3+
You want to generate creative and impactful research ideas that can be feasibly investigated with the code provided.
4+
Be critical and realistic in your assessments.
5+
6+
idea_first_prompt: |
7+
{task_description}
8+
<experiment.py>
9+
{code}
10+
</experiment.py>
11+
12+
Here are the ideas that you have already generated:
13+
14+
'''
15+
{prev_ideas_string}
16+
'''
17+
18+
Come up with the next impactful and creative idea for research experiments and directions you can feasibly investigate with the code provided.
19+
Note that you will not have access to any additional resources or datasets.
20+
Make sure any idea is not overfit the specific training dataset or model, and has wider significance.
21+
22+
Respond in the following format:
23+
24+
THOUGHT:
25+
<THOUGHT>
26+
27+
NEW IDEA JSON:
28+
```json
29+
<JSON>
30+
```
31+
32+
In <THOUGHT>, first briefly discuss your intuitions and motivations for the idea. Detail your high-level plan, necessary design choices and ideal outcomes of the experiments. Justify how the idea is different from the existing ones.
33+
34+
In <JSON>, provide the new idea in JSON format with the following fields:
35+
- "Name": A shortened descriptor of the idea. Lowercase, no spaces, underscores allowed.
36+
- "Title": A title for the idea, will be used for the report writing.
37+
- "Experiment": An outline of the implementation. E.g. which functions need to be added or modified, how results will be obtained, ...
38+
- "Interestingness": A rating from 1 to 10 (lowest to highest).
39+
- "Feasibility": A rating from 1 to 10 (lowest to highest).
40+
- "Novelty": A rating from 1 to 10 (lowest to highest).
41+
42+
Be cautious and realistic on your ratings.
43+
This JSON will be automatically parsed, so ensure the format is precise.
44+
You will have {num_reflections} rounds to iterate on the idea, but do not need to use them all.
45+
46+
idea_reflection_prompt: |
47+
Round {current_round}/{num_reflections}.
48+
In your thoughts, first carefully consider the quality, novelty, and feasibility of the idea you just created.
49+
Include any other factors that you think are important in evaluating the idea.
50+
Ensure the idea is clear and concise, and the JSON is the correct format.
51+
Do not make things overly complicated.
52+
In the next attempt, try and refine and improve your idea.
53+
Stick to the spirit of the original idea unless there are glaring issues.
54+
55+
Respond in the same format as before:
56+
THOUGHT:
57+
<THOUGHT>
58+
59+
NEW IDEA JSON:
60+
```json
61+
<JSON>
62+
```
63+
64+
If there is nothing to improve, simply repeat the previous JSON EXACTLY after the thought and include "I am done" at the end of the thoughts but before the JSON.
65+
ONLY INCLUDE "I am done" IF YOU ARE MAKING NO MORE CHANGES.
66+
67+
novelty_system_prompt: |
68+
You are an ambitious AI PhD student who is looking to publish a paper that will contribute significantly to the field.
69+
You have an idea and you want to check if it is novel or not. I.e., not overlapping significantly with existing literature or already well explored.
70+
Be a harsh critic for novelty, ensure there is a sufficient contribution in the idea for a new conference or workshop paper.
71+
You will be given access to the Semantic Scholar API, which you may use to survey the literature and find relevant papers to help you make your decision.
72+
The top 10 results for any search query will be presented to you with the abstracts.
73+
74+
You will be given {num_rounds} to decide on the paper, but you do not need to use them all.
75+
At any round, you may exit early and decide on the novelty of the idea.
76+
Decide a paper idea is novel if after sufficient searching, you have not found a paper that significantly overlaps with your idea.
77+
Decide a paper idea is not novel, if you have found a paper that significantly overlaps with your idea.
78+
79+
{task_description}
80+
<experiment.py>
81+
{code}
82+
</experiment.py>
83+
84+
novelty_prompt: |
85+
Round {current_round}/{num_rounds}.
86+
You have this idea:
87+
88+
"""
89+
{idea}
90+
"""
91+
92+
The results of the last query are (empty on first round):
93+
"""
94+
{last_query_results}
95+
"""
96+
97+
Respond in the following format:
98+
99+
THOUGHT:
100+
<THOUGHT>
101+
102+
RESPONSE:
103+
```json
104+
<JSON>
105+
```
106+
107+
In <THOUGHT>, first briefly reason over the idea and identify any query that could help you make your decision.
108+
If you have made your decision, add "Decision made: novel." or "Decision made: not novel." to your thoughts.
109+
110+
In <JSON>, respond in JSON format with ONLY the following field:
111+
- "Query": An optional search query to search the literature (e.g. attention is all you need). You must make a query if you have not decided this round.
112+
113+
A query will work best if you are able to recall the exact name of the paper you are looking for, or the authors.
114+
This JSON will be automatically parsed, so ensure the format is precise.

0 commit comments

Comments
 (0)