GH-28859: [Doc][Python] Use only code-block directive and set up doctest for the python user guide

[AlenkaF]

Rationale for this change

In many places in the Python User Guide the code exampels are written with IPython directive (elsewhere code-block is used). IPython directives are converted to IPython format (In and Out during the doc build). This can lead to slower builds.

What changes are included in this PR?

IPython directives are converted to runnable code-block (with >>> and ...) and pytest doctest support for .rst files is added to the conda-python-docs CI job. This means the code in the Python User Guide is tested separately to the building of the documentation.

Are these changes tested?

Yes, with the CI.

Are there any user-facing changes?

Changes to the Python User Guide examples will have to be tested with pytest --doctest-glob='*.rst' docs/source/python/file.rst

• GitHub Issue: [Doc][Python] The use of IPython directive or doctest code blocks in the python user guide #28859

[AlenkaF]

Converting this PR to draft till I figure out what would be the best way to run RST doctest on 3.12 Sphinx Documentation CI job and not on the Python 3.10 Sphinx & Numpydoc.

[AlenkaF]

@raulcd the main pain point why AMD64 Conda Python 3.10 Sphinx & Numpydoc fails and AMD64 Conda Python 3.12 Sphinx Documentation succeeds is the Python version and the use of datetime.UTC which was only added in Python 3.11, see https://docs.python.org/3/library/datetime.html#datetime.UTC.

I think the easiest solution would be to run Sphinx & Numpydoc on Python 3.11, or even Python 3.12 (I am not aware of any reason we would need the olderst Python version we support here. Sphinx Documentation runs on docs changes only while Sphinx & Numpydoc runs on any Python or C++ changes and validates the docstrings).

[raulcd]

Thanks for checking that @AlenkaF ! So currently we are providing a snippet on our documentation:

.. ipython:: python
    :okexcept:

    import datetime

    current_year = datetime.datetime.now(datetime.UTC).year
    for table_chunk in birthdays_dataset.to_batches():
        print("AGES", pc.subtract(current_year, table_chunk["years"]))

that will fail for some users as we are still supporting Python 3.10, right? Is it worth for the example to add the datetime.UTC? Should we just use for the example: current_year = datetime.datetime.now().year
Or maybe add a comment with a note?

I am ok to just bump the Python version of the job but we probably should not provide examples that will fail on some of the supported versions.

[AlenkaF]

Yeah, you are right. Changing to datetime.datetime.now().year or even datetime.datetime.now(datetime.timezone.utc) makes much more sense! Will update 👍

[AlenkaF]

Ha ha, the example would fail anyways as the year changed in the meantime 🤣
Probably it is best to just hardcode it.

[raulcd]

Ha ha, the example would fail anyways as the year changed in the meantime 🤣 Probably it is best to just hardcode it.

Yes, we don't want to have to update this every year because the data changes 😄