Automated Integration Test Goldens Update from CI #5751
Conversation
…633-b54d-3b4fa9167e69)
Summary of ChangesHello @datacommons-robot-author, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request automatically updates golden files for integration tests, reflecting recent changes in the system's query processing and statistical variable matching logic. The updates ensure that the test suite accurately represents the current behavior of the detection API and chart generation, maintaining consistency between expected and actual outputs. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request contains automated updates to integration test golden files. While many changes are minor adjustments to scores, which is expected, I've identified two issues that need attention. There is a critical regression in the numberofpoorhispanicwomenwithphd test case where detection now fails completely. Additionally, the compareobesityvs.poverty test case appears to be using an invalid configuration for the embeddings index. These issues suggest potential problems in the underlying code or the test generation process that should be resolved.
| "query_with_places_removed": "", | ||
| "sv_matching": { | ||
| "CosineScore": [ | ||
| 0.79858, | ||
| 0.78833, | ||
| 0.77635, | ||
| 0.77406, | ||
| 0.74103, | ||
| 0.73347, | ||
| 0.72283, | ||
| 0.72257, | ||
| 0.69955, | ||
| 0.6995, | ||
| 0.69762, | ||
| 0.69743, | ||
| 0.696, | ||
| 0.69534, | ||
| 0.69452, | ||
| 0.69396, | ||
| 0.69354, | ||
| 0.69084, | ||
| 0.6907, | ||
| 0.68967, | ||
| 0.68895, | ||
| 0.68862, | ||
| 0.68633, | ||
| 0.68592, | ||
| 0.68275, | ||
| 0.68221, | ||
| 0.68156, | ||
| 0.67987, | ||
| 0.67686, | ||
| 0.67631, | ||
| 0.67515, | ||
| 0.6745, | ||
| 0.67257, | ||
| 0.6718, | ||
| 0.66917, | ||
| 0.669, | ||
| 0.66822, | ||
| 0.66781, | ||
| 0.66507, | ||
| 0.66487 | ||
| ], | ||
| "MultiSV": { | ||
| "Candidates": [ | ||
| { | ||
| "AggCosineScore": 0.8315, | ||
| "DelimBased": false, | ||
| "Parts": [ | ||
| { | ||
| "CosineScore": [ | ||
| 0.83118, | ||
| 0.831, | ||
| 0.82552, | ||
| 0.81955, | ||
| 0.81456, | ||
| 0.80907, | ||
| 0.80864, | ||
| 0.80744, | ||
| 0.8003, | ||
| 0.79858, | ||
| 0.78782 | ||
| ], | ||
| "QueryPart": "number of poor hispanic", | ||
| "SV": [ | ||
| "Count_Person_BelowPovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_HispanicOrLatino", | ||
| "Count_Person_Female_AbovePovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_AbovePovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Household_WithoutFoodStampsInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_Male_AbovePovertyLevelInThePast12Months_BlackOrAfricanAmericanAlone", | ||
| "Count_Person_Male_BelowPovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_Female_BelowPovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_NoHealthInsurance_HispanicOrLatino", | ||
| "Count_Person_WithDisability_HispanicOrLatino", | ||
| "Count_Person_15OrMoreYears_Separated_HispanicOrLatino" | ||
| ] | ||
| }, | ||
| { | ||
| "CosineScore": [ | ||
| 0.83183, | ||
| 0.80299 | ||
| ], | ||
| "QueryPart": "women phd", | ||
| "SV": [ | ||
| "Count_Person_25OrMoreYears_EducationalAttainmentDoctorateDegree_Female", | ||
| "Count_Person_25OrMoreYears_Female_DoctorateDegree_AsFractionOf_Count_Person_25OrMoreYears_Female" | ||
| ] | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "AggCosineScore": 0.8259, | ||
| "DelimBased": false, | ||
| "Parts": [ | ||
| { | ||
| "CosineScore": [ | ||
| 0.83667, | ||
| 0.79727 | ||
| ], | ||
| "QueryPart": "number of poor", | ||
| "SV": [ | ||
| "dc/topic/Poverty", | ||
| "Count_Person_Rural_BelowPovertyLevelInThePast12Months" | ||
| ] | ||
| }, | ||
| { | ||
| "CosineScore": [ | ||
| 0.81515, | ||
| 0.7886, | ||
| 0.77756, | ||
| 0.77521 | ||
| ], | ||
| "QueryPart": "hispanic women phd", | ||
| "SV": [ | ||
| "dc/06f0jf8xvzw4f", | ||
| "Count_Person_Female_HispanicOrLatino", | ||
| "dc/3w039ndqy7qv1", | ||
| "dc/topic/HispanicOrLatinoFemalePopulationByAge" | ||
| ] | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "AggCosineScore": 0.8071, | ||
| "DelimBased": false, | ||
| "Parts": [ | ||
| { | ||
| "CosineScore": [ | ||
| 0.84959, | ||
| 0.8335 | ||
| ], | ||
| "QueryPart": "number of poor hispanic women", | ||
| "SV": [ | ||
| "Count_Person_Female_BelowPovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_Female_HispanicOrLatino" | ||
| ] | ||
| }, | ||
| { | ||
| "CosineScore": [ | ||
| 0.7647, | ||
| 0.74767, | ||
| 0.73771, | ||
| 0.73628, | ||
| 0.73059 | ||
| ], | ||
| "QueryPart": "phd", | ||
| "SV": [ | ||
| "Count_Person_EducationalAttainmentDoctorateDegree", | ||
| "Count_Person_25OrMoreYears_EducationalAttainmentDoctorateDegree_Female", | ||
| "Count_Person_25OrMoreYears_DoctorateDegree_AsFractionOf_Count_Person_25OrMoreYears", | ||
| "Count_Person_25OrMoreYears_EducationalAttainmentDoctorateDegree_Male", | ||
| "Count_Person_25OrMoreYears_Female_DoctorateDegree_AsFractionOf_Count_Person_25OrMoreYears_Female" | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| }, | ||
| "CosineScore": [], | ||
| "MultiSV": {}, | ||
| "Query": "number of poor hispanic women with phd", | ||
| "SV": [ | ||
| "dc/06f0jf8xvzw4f", | ||
| "Count_Person_Female_BelowPovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_Female_HispanicOrLatino", | ||
| "dc/3w039ndqy7qv1", | ||
| "dc/topic/HispanicOrLatinoFemalePopulationByAge", | ||
| "dc/9cqv67nn7pn1b", | ||
| "Median_Age_Person_Female_HispanicOrLatino", | ||
| "Count_Person_25OrMoreYears_EducationalAttainmentDoctorateDegree_Female", | ||
| "Count_Person_Female_AbovePovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_BelowPovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_15OrMoreYears_Widowed_HispanicOrLatino", | ||
| "Count_Person_15OrMoreYears_Divorced_HispanicOrLatino", | ||
| "Count_Household_HouseholderRaceHispanicOrLatino_SingleMotherFamilyHousehold", | ||
| "Count_Person_Female_NotHispanicOrLatino", | ||
| "Count_Student_HispanicOrLatino", | ||
| "Count_Person_25OrMoreYears_Female_DoctorateDegree_AsFractionOf_Count_Person_25OrMoreYears_Female", | ||
| "dc/0jtctjm33mgh1", | ||
| "Count_Person_WithDisability_HispanicOrLatino", | ||
| "Count_Person_15OrMoreYears_MarriedAndNotSeparated_HispanicOrLatino", | ||
| "Count_Person_HispanicOrLatino_ResidesInCollegeOrUniversityStudentHousing", | ||
| "Count_Household_WithoutFoodStampsInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_HispanicOrLatino", | ||
| "Count_Person_AbovePovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_Female_BelowPovertyLevelInThePast12Months_TwoOrMoreRaces", | ||
| "Count_Person_Male_AbovePovertyLevelInThePast12Months_BlackOrAfricanAmericanAlone", | ||
| "Count_Person_15OrMoreYears_NeverMarried_HispanicOrLatino", | ||
| "Count_Person_Male_BelowPovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_Female_BelowPovertyLevelInThePast12Months", | ||
| "dc/topic/PovertyByGender", | ||
| "Count_Person_Female_AbovePovertyLevelInThePast12Months_TwoOrMoreRaces", | ||
| "Count_Person_HispanicOrLatino_ResidesInNursingFacilities", | ||
| "Count_Person_NoHealthInsurance_HispanicOrLatino", | ||
| "dc/5hc4etrfyj9qg", | ||
| "Count_Person_Male_HispanicOrLatino", | ||
| "Count_Person_Female_BelowPovertyLevelInThePast12Months_WhiteAlone", | ||
| "dc/hyfn2tlyz48lb", | ||
| "Count_Person_25OrMoreYears_EducationalAttainmentSomeCollegeLessThan1Year_Female", | ||
| "Count_Person_25OrMoreYears_EducationalAttainmentBachelorsDegreeOrHigher_Female", | ||
| "Count_Person_Female_AbovePovertyLevelInThePast12Months_WhiteAlone", | ||
| "dc/epw58ne8mytn5" | ||
| ] | ||
| "SV": [] |
There was a problem hiding this comment.
These changes indicate a significant regression in query detection. The sv_matching object is now empty, meaning no statistical variables were detected for the query "number of poor hispanic women with phd". The previous version correctly identified relevant variables. This update effectively breaks the test case for this query and should be investigated. It seems the detection has fallen back from a functioning LLM-based approach to a failing heuristic one.
| "sv_detection_query_index_types": [ | ||
| "base_uae_mem" | ||
| ], | ||
| "sv_detection_query_input": "compare obesity vs poverty", | ||
| "sv_detection_query_stop_words_removal": "obesity poverty" | ||
| "" | ||
| ] |
There was a problem hiding this comment.
The value for sv_detection_query_index_types is [""]. An empty string is likely not a valid embeddings index type and could lead to silent failures or unexpected behavior in statistical variable detection. This might indicate an issue with the test configuration that generated this golden file. Please ensure a valid index name (e.g., "base_uae_mem") is used.
1 participant
This pull request updates the golden files automatically via Cloud Build. Please review the changes carefully. Cloud Build Log