Orientation questions

[bernd-wechner]

Quite excited to bump into this. Been pottering on ways to manage my library for years with many false starts and patch successes. And community produces so much more and better outcomes than going it alone (at least for busy folk with fingers in a hundred pies and short on time across them all).

I have a couple of orientation questions though not satisfied from my read of docs thus far (if for any reason I missed any of this in the docs be patient with me, I did try - I'm not into wasting anyone's time with questions I can answer myself).

I'm curious about:

1. The database. It's SQLite I imagine. Could beets be pointed at an existing rdbms (postgresql?)

2. Scaleability. If your music library is multiple hundred of thousand of tracks in size (who me?) most things I've tried grind to halt and fail to be useful. Beets?

3. Workflow. I keep my library on a NAS, and I keep a backup on another server, and I have an incoming folder on yet another machine where are parked albums yet to process. I see two different jobs I'd like covered

   1. tidying up the existing library, applying some rules to tags, I'm specifically interest in:
      1. adding an AlbumArtist tag to all files based on sensible inference from context,
      2. fixing the case of titles across the board (harder than you'd think given the propensity Mc, Roman Numerals and proper nouns etc, it's something I figure AI could really play a role in slowly,
      3. Adding question marks to titles that are questions. Again harder than I ever thought ti might be and not a bad job for AI nowadays.
      • Some of that can be fixed using MusicBrainz metadata if it's clean, as the vast majority of it is already tagged.
   2. Cleaning incoming albums, tagging, getting cover images, and then moving them into the library proper.

I'm more than happy to try beets out and if and when opportunity or need presents could imagine contributing if I felt any changes I needed were generic (likely to be of broader appeal). In the mean-time I love that there are discussions and there's a chance I might get some reflections on these goals of mine.

[semohr]

Really great to hear your thoughts and goal, they resonate with a lot of us here, and it sounds like you've already put some serious consideration into your workflow.

To touch on your questions from my perspective:

1. Database: Beets is indeed tightly coupled with SQLite. At the moment, there’s no support for other RDBMS like PostgreSQL. Beets doesn’t use a general-purpose ORM like SQLAlchemy, so switching out the backend isn’t currently feasible without major surgery (I think beets even predates many python ORMs)
2. Scalability: Beets scales well, even with large libraries. Since it mainly interacts with the database and filesystem without loading everything into memory, it's efficient. I’ve seen reports of folks managing terabytes of music and hundreds of thousands of tracks with it. Personally, my library is around 300GB, and performance hasn’t been (too much of an) issue.
3. Workflow: What you're describing is a very common setup, and beets is quite well-suited to handling it imo especially with the right mix of plugins and scripting. Here's how I see your two main jobs:
   • Tidying the existing library: Totally doable. You can write custom queries and use plugins or scripts to enforce tag cleanup. For things like:
     • Inferring albumartist, beets can pull from different metadatasource (mb, spotify, discogs) or even filenames.
     • Fixing title case and punctuation might gets tricky fast. I don't think there is an AI plugin for this right now but you might be able to create one if it is needed. To be honest tho using a metadatasource is good enought most of the time if you are not primarly collecting dupes or bootlegs.
   • Cleaning new albums and importing: Beets was built for this. The import process can be automated and customized a lot, tagging, fetching artwork, renaming and moving files into your main library structure. To shamelesly plug my own project, you might also want to have a look at beets-flask, which adds a web frontend for importing new albums.

All in all, you’re in the right place, and I think beets could cover most of what you’re after, either out of the box or with a few custom tweaks. And if you end up building something reusable, the community would definitely welcome contributions and would love to have you onboard.

Best,
Sebastian

[RollingStar]

Adding to the above.

The beets import sequence is along the lines of:

1. Match a given file to a specific known song. This uses musicbrainz by default but there are plugins for e.g. Discogs.
2. Once matched, or if matching is skipped ("import as-is"), it goes in the beets sqlite database with fields like artist, album, title, length etc.
3. The metadata in the beets database is used to create a file path. you can change this whenever you want but you have to run beet mv and there are caveats.

The big caveat is to my knowledge, stock beets out of the box doesn't do a great job with 'extra files'. everything outside of album art gets left wherever you imported it. And then you have files matching to an album with no hope of joining the extra files back to the album. we have plugins that mitigate this but you must configure them BEFORE your initial import of the folder. i.e. you don't want to import 1000 albums and then go back and decide "wait, I want the other files (cuesheets, readme.txt, other scans?) that go with JS Bach - Goldberg Variations".

I think the above problem is less bad if we store the original import path of every file, but I don't think we do that out of the box either. IIRC there was a plugin for that ages ago.

The other caveat is beet mv necessarily involves a filesystem operation for every file that changes paths. Even on an SSD this is a slight overhead. on an HDD it can be non trivial when you're talking about 300k files.

1. Beets has had a nonzero amount of file deletion bugs and subjectively "unexpected" file deletion situations. Back up your files if at all possible.
2. consider running on linux or macOS. It just generally makes stuff easier, especially when you get involved with plugins. WSL may be an option but I haven't heard anyone using it for beets.
3. Start slow with 1-50 albums. move slowly, work in interactive mode (--timid?) and watch what beets is doing. maybe make a backup of the files you're working on. Over time you will get an opinion of how you want your filesystem to look and you will set more path options. Eventually you will know and trust beets enough to run it on all your files.
4. If you can somehow recover more data from your files then the autotagging gets easier. We don't do this currently, but it's theoretically possible to grab barcodes from album scans. Or use image recognition to do OCR or finding similar album covers.
5. beets defaults are generally for 'music nerds' not 'music obsessives'. If you have 300k tracks you might have specific opinions about how the beets pipeline should work. our defaults work pretty good for people with a few hundred or thousand albums, but might need tweaking if you have multiple versions of the same album or multiple albums with the same title (Weezer "Blue" vs. "Green"). You might need to flex the disambiguation, autotag, automatch, web db (musicbrainz), acoustic match (chroma) options. matching at the release level (2009 remaster of Thriller) rather than the release group level (any release of Thriller) is a hard computer science problem that comes down to the quality of your input data (how well tagged are your files), the quality of your algorithm, and the quality of web databases (musicbrainz). in almost 10 years of being aware of beets, I have come to one conclusion: In almost all cases, normal people should care about matching at the release group level, not the release level. we have few if any use cases that demonstrate the power of having metadata at the release level. If you don't have multiple releases for the same group, I struggle to see the use case. beets is not really built for matching at the group level, but maybe consider being more liberal with 'as is' imports or just autotagging and not caring if it tags to the vinyl or CD release. just throw a huge penalty on missing tracks and it should at least get the track list correct.
6. if you can store the input paths (not sure how hard this is), do it.
7. hot take: chromaprint should be enabled by default.

[RollingStar]

Adding to the above.

The beets import sequence is along the lines of:

1. Match a given file to a specific known song. This uses musicbrainz by default but there are plugins for e.g. Discogs.
2. Once matched, or if matching is skipped ("import as-is"), it goes in the beets sqlite database with fields like artist, album, title, length etc.
3. The metadata in the beets database is used to create a file path. you can change this whenever you want but you have to run beet mv and there are caveats.

The big caveat is to my knowledge, stock beets out of the box doesn't do a great job with 'extra files'. everything outside of album art gets left wherever you imported it. And then you have files matching to an album with no hope of joining the extra files back to the album. we have plugins that mitigate this but you must configure them BEFORE your initial import of the folder. i.e. you don't want to import 1000 albums and then go back and decide "wait, I want the other files (cuesheets, readme.txt, other scans?) that go with JS Bach - Goldberg Variations".

I think the above problem is less bad if we store the original import path of every file, but I don't think we do that out of the box either. IIRC there was a plugin for that ages ago.

The other caveat is beet mv necessarily involves a filesystem operation for every file that changes paths. Even on an SSD this is a slight overhead. on an HDD it can be non trivial.

1. Beets has had a nonzero amount of file deletion bugs and subjectively "unexpected" file deletion situations. Back up your files if at all possible.
2. consider running on linux or macOS. It just generally makes stuff easier, especially when you get involved with plugins. WSL may be an option but I haven't heard anyone using it for beets.
3. Start slow with 1-50 albums. move slowly, work in interactive mode (--timid?) and watch what beets is doing. maybe make a backup of the files you're working on. Over time you will get an opinion of how you want your filesystem to look and you will set more path options. Eventually you will know and trust beets enough to run it on all your files.
4. If you can somehow recover more data from your files then the autotagging gets easier. We don't do this currently, but it's theoretically possible to grab barcodes from album scans. Or use image recognition to do OCR or finding similar album covers.
5. beets defaults are generally for 'music nerds' not 'music obsessives'. If you have 300k tracks you might have specific opinions about how the beets pipeline should work. our defaults work pretty good for people with a few hundred or thousand albums, but might need tweaking if you have multiple versions of the same album or multiple albums with the same title (Weezer "Blue" vs. "Green"). You might need to flex the disambiguation, autotag, automatch, web db (musicbrainz), acoustic match (chroma) options. matching at the release level (2009 remaster of Thriller) rather than the release group level (any release of Thriller) is a hard computer science problem that comes down to the quality of your input data (how well tagged are your files), the quality of your algorithm, and the quality of web databases (musicbrainz). in almost 10 years of being aware of beets, I have come to one conclusion: In almost all cases, normal people should care about matching at the release group level, not the release level. we have few if any use cases that demonstrate the power of having metadata at the release level. If you don't have multiple releases for the same group, I struggle to see the use case. beets is not really built for matching at the group level, but maybe consider being more liberal with 'as is' imports or just autotagging and not caring if it tags to the vinyl or CD release. just throw a huge penalty on missing tracks and it should at least get the track list correct.
6. if you can store the input paths (not sure how hard this is), do it.
7. hot take: chromaprint should be enabled by default.

[bernd-wechner]

Thanks for the feedback. Very much appreciate. Doing my first trials now and also have it set up in an IDE for tracing through code paths if I feel that way inclined (which I rapidly do but will start empirically).

Interesting terminology, music nerd and music obsessive, I wouldn't have used either label for myself, but I do like consistency and I have preferred tagging and naming layout to respect (that's a personality trait some describe with eye to consistency and quality, others as OCD, horses for courses).

I got the naming fairly right with the path: config, but the tags need work. Here are some inital findings:

This config:

set_fields:
year: '$year'
track: '$track'

I should have hoped ensured that the default which I see in my test album import of populating the year tag with a day resolution date and the tack tag with $track/$tracktotal is change my preference to year resolution only in the year tag and track only in the track tag.

But no, didn't seem to have an impact. Other notes I made:

1. Cover image not saved as a file (added a thumbnail to ID3 tags of each song)
2. If I remove the file from the folder from the target direct and reimport two behavours could improve:
   a. The "could not get filesize: [Errno 2] No such file or directory:" for each missing file could be handled gracefully and not shown,
   b. The disambiguator which ends up added to the album name inht e path could be skipped.
   c. I imagine an option that requests "If it's in the database but does not exist on disk, remove it from the database" for anyone wanting to implciily remove things that way. Just a thought. Not a biggie as I can/should remove the album before reimporting.
3. I can't find a way to specify conditional paths for single disc and multidisc albums. In my GUI taggers I do just that typically (have done for decades) producing "tt - title.ext" for singe disc files and "d - tt - title.ext" for files in a multidisc album.
4. I can't find a way to ask for a conditional write of the Disc Number tag, only for multidisc albums and then only the disc number not the count.

Small things, but the sort that will see me looking at the code next to see I can spot easy ways to achieve what I wnat, or consider adding options and PRs.

On the whole, quite nice a framework, and some very lovely behaviours as a default. I am also happy to contribute to docs on readthedocs as I find time to capture the questions I had and got answers for. ONce I work out how I guess.

[JOJ0]

look at fetchart/embedart plugins

best you let beets handle all file operations, it's just the way it's designed. beet rm -d, beet mv,

though, beet update removes files if not found anymore IIRC

I use the inline plugin (see plugins list in docs) to do conditional track number formatting. Something like this:

item_fields:
    disc_and_track: u'%02i.%02i' % (disc, track) if
                    disctotal > 1 else u'%02i' % (track)

Use in path format as$disc_and_track