Skip to content

Qualtrics reporting #855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 23 commits into
base: dev
Choose a base branch
from
Draft

Qualtrics reporting #855

wants to merge 23 commits into from

Conversation

luxaritas
Copy link
Member

@luxaritas luxaritas commented Sep 25, 2025 •
edited
Loading

Summary

Implements Qualtrics reporting for Ribotree SAL/HAI seed grant user study

  • Adds postmessage reporter which filters for relevant events, tied to the qualtrics domain we'll use
  • Adds qualtrics-report feature flag to enable reporter and disable sending the solution to the server
  • Add reporting for script constraint results (as we want to track how the degscore changes over time)
  • Add new timer constraint which prevents the puzzle from solving until the time has elapsed. For this study we want the user to continue playing the puzzle for a certain amount of time, then auto-complete (this will be combined with soft constraints)
  • Supports suppressing the checkmark/glow/sound when a constraint is satisfied as it doesn't make sense for the timer constraint
  • Support soft constraints for challenge puzzles

Implementation Notes

  • We now always create the outline texture even if outline is disabled since it's used for positioning eg the stattext. It would maybe be better to decouple, but this seems to be an innocuous change to maintain the status quo.
  • We selectively only re-evaluate the timer constraint if that is the only part of the state that has changed. This is a small performance benefit, but mostly this is to avoid re-firing eternascripts which may do who-knows what (rerunning this often could cause creating unknown regressions and setting a bad precedent for authors)
  • We split off the last part of updateScore that updates constraints and possibly submits because calling the entirety of updatescore would cause a specbox update, which triggers a poseop, which causes a flicker of the async text. Also this avoids the performance penalty of all the other irrelevant state sync stuff.

Testing

Created puzzle on dev with degscore and timer constraints, verifying it solves correctly and degscore constraint info logs via console reporter as expected. Also verified simple hairpin solves correctly

luxaritas added 23 commits August 30, 2025 12:21
@luxaritas
WIP
1e7fcbc
@luxaritas
UI observability
f837b3b
@luxaritas
Restore mutation tracking
6a7baef
@luxaritas
More o11y events
39b2383
@luxaritas
Observability streaming handlers, postmessage output, event filtering
121aebd
@luxaritas
Merge move tracking into o11y
b1404ae
@luxaritas
abstract class --> interface, console reporter for debugging
e96c741
@luxaritas
Tweak log string
f65c8f4
@luxaritas
Prevent toolbaraction events from being double-fired when fired on cl…
b8db367 
…ones, which propogate their events to the original button
@luxaritas
Fix folder and marker layer firing on puzzle setup, fix folder event …
a3bab07 
…detail to use name instead of object
@luxaritas
o11y events for mode changes
330c845
@luxaritas
Merge branch 'dev' into feat/observability
3e0ee99
@luxaritas
Add qualtrics reporter
4fe6934
@luxaritas
Fix broken return from script callbacks
3260756
@luxaritas
Merge branch 'feat/observability' into feat/salhai-report
0cdad67
@luxaritas
Add recording to scriptconstraint results
e410c5e
@luxaritas
Timer constraint
cee5d7f
@luxaritas
Record solution submission, skip upload for qualtrics report
2ebd321
@luxaritas
Trigger updates for timer constraint
ea706e1
@luxaritas
Avoid reevaluating constraints due to only time update
5c718a6 
In particular we want to avoid re-running eternascript constraints,
which could be doing who knows what. In addition to unknown expectations
they may have around only being called on certain user interactions,
we also want to avoid setting an expectation that they'll be called
regularly because that seems ripe for poor author decisions and
later breakakge if we change something
@luxaritas
Prevent async text flashing
65ccc79 
This was due to two things:
1) Using the opqueue to trigger updatescore
2) In updatescore, we also use the opqueue for updating the melting
   point and dot plot for the specbox

We only actually need to recheck constraints, so this also is a bit of
an optimization
@luxaritas
Supress glow and sound for timer constraint satisfied state change
742ccfe
@luxaritas
Support soft constraints for non-experimental puzzles
631c21a
@luxaritas
Copy link
Member Author

This is stacked on #852

@luxaritas
Copy link
Member Author

For a stack diff: feat/observability...feat/salhai-report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@luxaritas