Archiving

[smartspot2]

Thread for discussion/ideas/designs for the archiving system. (Link to the current project board).

[smartspot2]

Some initial thoughts:

• We need a system to efficiently and easily archive semesters of data, so that we aren't constantly wiping the database manually every semester
• This also means that we'd need to change the way our database is structured, and add a way to differentiate between semesters
• This further means that all queries and database accesses will need to filter by semester—we need a solution for this as well, because manually changing all queries is not efficient

Some initial ideas:

• It'd be nice to have a UI/page/button in the admin interface to set the current semester
• There is a way to set a default filter for a model's objects attribute (relevant stackoverflow post); we can use that to simplify the process
• There are a few options in how we want to assign semesters:
  • We can assign the Course model and the User model a semester field, set to some number (or string, ex "fa21", "sp22", etc.?) for the current semester
  • We can create a new Semester model and have Course and User have a ForeignKey field pointing to the current Semester object
• In any of the options though, we can add an is_current method (or property?), simply returning a boolean for whether or not the current object is associated with the current semester
  • This would be specialized for each model, and we can share the object manager (as mentioned prior) between all models to simply filter by is_current
  • The name of the method can be changed to be a little more descriptive on what it's responsible for if necessary
  • We can still have an objects_all field that does not use the filtered manager, in case we do want to query through all archived data

[smartspot2]

In regard to having a setting on the admin interface to set the current semester, upon further research, this seems to be a little bit more complicated than I originally thought.

It's bad to store and edit settings that we'd be editing a lot in settings.py, and the only other alternative would be to store a key/value pair in the database as a normal database entry. One option would be to make our own Setting model and use that; the downside is that it's quite a lot of code that would need to be written (creating the model, adding an admin page, etc.) to just have one setting for the current semester, but the upside is that we'd have a good foundation built, in case we would like to have more configurable site-wide settings in the future. Going this route would also cause more database accesses when we check whether an object is associated with a current semester.

There are also several Django libraries already out there for dynamic settings management (django-dbsettings, django-constance, etc.; more here), but these are generally a lot more heavy-weight than we'd need in our case—especially if we only have one configurable setting at the moment. Of course, if we're planning to extend this functionality in the future and add a lot more settings, then this could be a good idea, but otherwise it's probably overkill.

If we do not care too much about having a setting exposed in the admin menu, there's a few alternatives. We could just write it into settings.py and leave it be, but this would mean that we'd need to make a commit to master every semester to change the current semester. We could also use an environment variable, but this would require somebody with Heroku access to change the environment variable on prod (but at the same time would be easier and probably better than making a commit).

---

In regard to checking whether a given object is associated with a current semester, we'd need to figure out a way to optimize the number of database calls we do.

At the moment, if we only associate Course and User with a semester object, I believe we'd have to go through three database accesses at most to do the check (ex. Attendance -> Student -> User, and one more if we want a Semester model). Handling a lot of objects at once, this could get problematic, unless we always prefetch the semester field/object, etc.; it's something to keep in mind as we design the archiving system.

[noloerino]

My 2 cents on this (take it with a grain of salt since I haven't interacted with the codebase for a while):

---

I don't think it makes semantic sense to put a foreign key to Semester on User. The User class is associated with all of an individual authentication information and such, which shouldn't change across semesters; if a user changes their email address or similar, then the User object should change accordingly anyway. You'd also run the risk of having Attendance -> Student -> User -> Semester being different from Attendance -> Student -> Section -> Course -> Semester, which would be unlikely but annoying to fix.

---

I'm not entirely sure what your use case for setting the "current semester" in the admin interface would bee. If the intent is to allow coords/exec to be able to view previous semesters' content, then a different semester (e.g. chosen through a dropdown on the frontend) would best be distinguished by an API query, and not modify anything in the database.

If your intent is instead to set the "active" semester (i.e. the one for which sections are currently happening), then I think you could probably accomplish this by either

• Maintaining some sort of ordering on Semester objects (e.g. fall 18 < spring 19 < fall 20, etc.), and the "current" semester is just the last object in this sort -- since we're never going to have more than like 10 semesters, this should be an acceptable cost
• Adding a date_start field on Semester objects, where the current semester is the most recent s such that s.date() < today.date() -- this approach would allow you to start entering data for future semesters, though I'm not sure why you would want to do that

Do let me know if I'm misunderstanding something here though.

---

In regard to checking whether a given object is associated with a current semester, we'd need to figure out a way to optimize the number of database calls we do.

I believe there's some trickery you can do with select_related to minimize foreign key lookups (there's probably a few instances in the codebase somewhere), but I'm not an expert on database stuff.