Support of including blog in the website search #2136

tim-schilling · 2025-07-23T22:14:25Z

The blog results should have a property of whether it is included in the search results. We should also limit the blogs that are searchable for a version of Django based on the support end. This will allow us to limit the inclusion of blog posts in the search based on the time the entry was created, keeping the search results relevant to that version of Django.

~~This is a WIP yet. I'm opening the draft PR to solicit early feedback on the approach. I'm thinking we'll re-use the DocumentationCategory.WEBSITE in place of "weblog".~~

tim-schilling · 2025-07-31T12:44:32Z

I was given tacit approval (lack of pushback) from the website WG to use this approach. I plan on working on this at the DjangoCon Africa sprints (Aug 14 & 15), but if anyone wants to complete it before that, please do!

tim-schilling · 2025-09-01T17:49:54Z

I'm going to try to clean up the commits shortly here.

…ter.

The blog results should have a property of whether it is included in the search results. We should also limit the blogs that are searchable for a version of Django based on the support end. This will allow us to limit the inclusion of blog posts in the search based on the time the entry was created, keeping the search results relevant to that version of Django.

tim-schilling · 2025-09-01T19:15:49Z

Cleaned up the commits. Please let me know if any other changes are necessary.

blog/migrations/0008_alter_entry_is_searchable.py

tim-schilling · 2025-09-03T19:02:46Z

docs/models.py

+                path = reverse_path(url_name, kwargs=kwargs)
+                request = RequestFactory().get(path, HTTP_HOST=www_host)
+                body = resolve(path).func(request).render().text
+                # Need to parse the body element.


Forgot this TODO yet.

@bmispelon do you have any good ideas here?

bmispelon

This is a fantastic PR, thanks for submitting it! It addresses a real issue that's been plaguing the site for a long time, and as far as PR go, it's a really high quality one, kudos 👏🏻

Of course because I'm me I'll always find something to complain about so I've left a few comments ranging from improvement suggestions to open questions.

I'll note that I have not tested the code locally (yet). I'm still working on getting the preview server up and running and I think this PR would be a great candidate for it.

blog/models.py

docs/tests/test_models.py

bmispelon · 2025-09-03T19:22:19Z

docs/tests/test_models.py

+        cls.document_index, cls.document_view, cls.document_detail = (
+            cls.release.documents.order_by("path")
+        )


This will break if/when more views are added to _sync_views_to_db(), right?

I'd suggest breaking the single test_document_url() method into three separate tests then.

Judging from the rest of the tests, it seems like there's an effort to keep them efficient. It may make more sense as two separate tests, one for the old document_url and then a new one for the DocumentationCategory.WEBSITE branch. Let me know if you disagree.

blog/models.py

bmispelon · 2025-09-03T19:50:55Z

docs/models.py

+            ]
+            if not www_hosts or not (www_host := www_hosts[0]):
+                return
+            synced_views = [


I think this should live in a constant at the top level of a module (I'll let you decide which module makes more sense 😁 )

If we're moving this ~~from the code~~ away from the usage logic, then we should use a dataclass so it's not relying on positional arguments. That gets a bit more verbose and likely could go in an entirely separate module.

May be side-stepped due to #2136 (comment)

bmispelon · 2025-09-03T19:59:13Z

docs/models.py

+                path = reverse_path(url_name, kwargs=kwargs)
+                request = RequestFactory().get(path, HTTP_HOST=www_host)
+                body = resolve(path).func(request).render().text


That's a lot of stuff going on in such a few lines: you make it look deceptively simple 😁

I'm curious: did you try other approaches before settling on this? I'm a bit worried that the whole www_host logic could be a bit brittle. If we're calling the view function directly anyway, can we skip setting the host on the request object?

This is definitely for a reason, but let me try once more and better document the weirdness.

Okay, so the simplest we can get it is to use get_template("aggregator/ecosystem.html").render(). However, that does sort of open us to the problem of the view (which is defined in the urls.py) of using a different template or implementation than expected. However, using this approach eliminates the need to get the host which eliminates having the search the ALLOWED_HOSTS setting. So I think it's worth it. Let me know what you think.

docs/models.py

Co-authored-by: Baptiste Mispelon <bmispelon@gmail.com>

This does mean that when the change is released to production, the Entry table will be locked entirely until it completes.

bmispelon

I like the move to a dataclass for SearchableView, nice one 👍🏻

I think we can go further and delegate the nitty-gritty details of getting a "document"'s URL and content to that class.

I've got grand ideas of generalizing this and having a collection of searchable/indexable models (sphinx document, blog, template view, flat page, ...) but that'd make the scope of this PR explode and I'd rather see this shipped. We can always iterate.

Other than that I left a question about the new support_end. I think we should be conservative with adding new fields and would like to explore reusing existing ones, to minimize the combinatorial explosion of possible model states. If we have to keep support_end, would it be possible for the existing DocumentRelease to get a sensible default assigned to them with a data migration (we have 100 of them in production, it'd be a shame if we had to go and back-populate them by hand).

Thanks again for your work on this!

bmispelon · 2025-09-08T12:57:49Z

docs/models.py

+    support_end = models.DateField(
+        null=True,
+        blank=True,
+        help_text="The end of support for this release of Django.",
+    )


Do we need a dedicated field for this, why couldn't we get the information from release.eol_date instead?

This is why we do code reviews. Let me get this switched over.

bmispelon · 2025-09-08T13:02:20Z

docs/models.py

+            absolute_url = reverse(searchable_view.url_name, host="www")
+            # This must match the template used for the url `community-ecosystem`
+            html = get_template("aggregator/ecosystem.html").render()
+            # Need to parse the body element.


Can we make absolute_url and html attributes (likely a property) of SearchableView? The name of the template should be an attribute of the SearchableView too and not be hardcoded here.

bmispelon · 2025-09-08T13:31:30Z

docs/search.py

    TOPICS = "topics", _("Using Django")
    HOWTO = "howto", _("How-to guides")
    RELEASE_NOTES = "releases", _("Release notes")
+    WEBSITE = "weblog", _("Django Website")


Do the breadcrumbs work for you in the first place? I don't think I've ever used that feature and it doesn't seem to work correctly for me locally: even for documents with two or more parents, the links for all the parents are all the same and go to the document's own page.

Not sure I'd want to start adding redirects, that seems like solving the problem at the wrong level to me.

pauloxnet · 2025-10-13T13:11:15Z

Can you review this PR based on this latest change in the search docs #2192

tim-schilling · 2025-10-13T14:56:59Z

@pauloxnet I don't see anything obvious that would cause a problem. As long as #2192 didn't change how the search data needs to be stored, then this PR won't be impacted. The hack that is this PR is to insert search-able data and point it to the blog or ecosystem page.

pauloxnet · 2025-10-13T15:00:32Z

I don't see anything obvious that would cause a problem. As long as #2192 didn't change how the search data needs to be stored, then this PR won't be impacted. The hack that is this PR is to insert search-able data and point it to the blog or ecosystem page.

I was indirectly suggesting using the approach outlined in the documentation, which involves delegating the generation of search vectors to a generated field. I'm sorry I don't have the time to do a more thorough review of this PR (I'm recuperating from my Django sprints). 😅

tim-schilling · 2025-10-13T15:17:18Z

@pauloxnet I think you can agree there's value in shipping functionality rather than perfecting the approach. If that suggestion is more of a, "hey we need to conform to this new standard", I can make the changes. If it's a "it'd be cool if it did this" type suggestion, I'm going to push for shipping something first.

pauloxnet · 2025-10-13T15:25:57Z

I think you can agree there's value in shipping functionality rather than perfecting the approach. If that suggestion is more of a, "hey we need to conform to this new standard", I can make the changes. If it's a "it'd be cool if it did this" type suggestion, I'm going to push for shipping something first.

My comment was, rather, I saw that you started this PR before the docs search improvement with GeneratedField, and, not having time to do a full review of the PR, I tried to point it out in this PR. No intent of perfectionism.

sarahboyce · 2025-10-15T09:33:20Z

docs/models.py

+        Sync the blog entries into search based on the release documents
+        support end date.
+        """
+        if self.lang != "en" or not self.release.eol_date:


Suggested change

if self.lang != "en" or not self.release.eol_date:

if self.lang != "en" or (self.release and not self.release.eol_date):

You need something like that as there are english "dev" docs which don't have a release object and eol date
Might be worth testing this case

Ah, my assumption was that any release had an EOL date set when it was created.

sarahboyce · 2025-10-15T09:49:47Z

blog/models.py

        ),
        default=False,
    )
+    is_searchable = models.BooleanField(


Do you have examples of blog entries you're expecting to be included and excluded? To me, I feel like everything should be searchable

No I haven't, but since this is searching the docs, not searching the site, I would imagine blog posts such as announcing board elections would not be searchable.

tim-schilling force-pushed the searchable-blog branch from 83133b1 to 4f9e4bf Compare July 24, 2025 12:29

tim-schilling force-pushed the searchable-blog branch from 4f9e4bf to 2bf75cd Compare September 1, 2025 16:07

tim-schilling marked this pull request as ready for review September 1, 2025 17:48

tim-schilling added 3 commits September 1, 2025 14:15

Added is_searchable and made published accept datetime cut-off parame…

03c5801

…ter.

Used Entry.get_absolute_url to encapsulate www host.

12c377e

tim-schilling force-pushed the searchable-blog branch from a9a4f8a to 837e808 Compare September 1, 2025 19:15

adamzap reviewed Sep 1, 2025

View reviewed changes

blog/migrations/0008_alter_entry_is_searchable.py Outdated Show resolved Hide resolved

tim-schilling commented Sep 3, 2025

View reviewed changes

bmispelon reviewed Sep 3, 2025

View reviewed changes

tim-schilling and others added 11 commits September 3, 2025 19:38

Switched to assertQuerySetEqual for documents existence check

5e22aee

Co-authored-by: Baptiste Mispelon <bmispelon@gmail.com>

Fixed is_searchable help_text. It's now English.

edea21c

Extracted get_search_config helper function.

e9d3d3b

Fixed docstring of _sync_views_to_db

f3c5a83

Switched the _sync helpers to use short-circuit if statements.

4fa01d6

Replaced WEBSITE.value usages with WEBSITE

021a57d

Switched to assertQuerySetEqual away from set comparisons.

e5a9452

Refactored DocumentUrlTests into two separate tests

bcffdee

Consolidated migrations to add is_searchable to one migration.

a7b3bbe

This does mean that when the change is released to production, the Entry table will be locked entirely until it completes.

Reworked _sync_views_to_db to almost be readable.

c1a61a9

Switched published argument to as_of

8e01cca

bmispelon reviewed Sep 8, 2025

View reviewed changes

tim-schilling added 4 commits September 17, 2025 16:36

Replaced DocumentRelease.support_end with Release.eol_date

629bc6d

Moved more logic into SearchableView.

c89240a

Renamed the key of DocumentationCategory.WEBSITE

606d381

Switched sync methods to use Document.objects.bulk_create()

396050c

sarahboyce reviewed Oct 15, 2025

View reviewed changes

	if self.lang != "en" or not self.release.eol_date:
	if self.lang != "en" or (self.release and not self.release.eol_date):

Uh oh!

Support of including blog in the website search #2136

Are you sure you want to change the base?

Support of including blog in the website search #2136

Conversation

tim-schilling commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tim-schilling commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tim-schilling commented Sep 1, 2025

Uh oh!

tim-schilling commented Sep 1, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bmispelon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tim-schilling Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tim-schilling Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bmispelon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pauloxnet commented Oct 13, 2025

Uh oh!

tim-schilling commented Oct 13, 2025

Uh oh!

pauloxnet commented Oct 13, 2025

Uh oh!

tim-schilling commented Oct 13, 2025

Uh oh!

pauloxnet commented Oct 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

tim-schilling commented Jul 23, 2025 •

edited

Loading

tim-schilling commented Jul 31, 2025 •

edited

Loading

tim-schilling Sep 4, 2025 •

edited

Loading

tim-schilling Sep 4, 2025 •

edited

Loading