HDXDSYS-2471 New non-HXL resource generation functions

documentation/index.md

@@ -54,6 +54,11 @@ The library has detailed API documentation which can be found in the menu at the
 
 
 ## Breaking Changes
+From 6.5.2, remove unused `generate_qc_resource_from_rows` method.
+`generate_resource_from_rows`, `generate_resource_from_iterable` and
+`download_and_generate_resource` are deprecated. They are replaced by
+`generate_resource` and `download_generate_resource`.
+
 From 6.5.0, files will not be uploaded to the HDX filestore if the hash and size have
 not changed, but if there are any resource metadata changes, except for last_modified,
 they will still take place.
@@ -821,26 +826,14 @@ dictionary. HEADERS is either a row number (rows start counting at 1), or the
 actual headers defined as a list of strings. If not set, all rows will be
 treated as containing values:
 
-    dataset.generate_resource_from_rows("FOLDER", "FILENAME", ROWS,
-                                        RESOURCE DATA, HEADERS, "ENCODING")
-
-A resource for the purpose of driving QuickCharts can be generated by taking
-ROWS, a list of dictionaries, and producing a cut down subset from it. HXLTAGS
-are added as the second row after the header. The reduction in rows is
-performed by only outputting the rows where COLUMN_NAME has a value in
-QC_IDENTIFIERS. Optionally the columns that are output can be limited by
-specifying them in HEADERS.
-
-    dataset.generate_qc_resource_from_rows("FOLDER", "FILENAME", ROWS,
-    RESOURCE DATA, HXLTAGS, "COLUMN_NAME", QC_IDENTIFIERS, HEADERS, "ENCODING")
+    dataset.generate_resource("FOLDER", "FILENAME", ROWS, RESOURCE DATA, HEADERS,
+                              COLUMNS, "FORMAT", "ENCODING", DATECOL or YEARCOL or
+                              DATE_FUNCTION)
 
-Building on these basic resource generation methods, there are more powerful
-ones `generate_resource_from_iterator` and `download_and_generate_resource`.
-
-A resource can be generated from a given list or tuple: HEADERS and an ITERATOR
-which can return rows in list, tuple or dictionary form. A mapping from headers
-to HXL hashtags, HXLTAGS, must be provided along with the FOLDER and FILENAME
-where the file will be generated for upload to the filestore. The dataset
+The first 4 parameters are mandatory, the rest are optional. A resource can be generated
+from a given list or tuple or other iterable. The method returns a tuple with a bool
+True is the resource was addeed and a dictionary of information. FOLDER and FILENAME
+specify where the file will be generated for upload to the filestore. The dataset
 time period can optionally be set by supplying DATECOL for looking up
 dates or YEARCOL for looking up years. DATECOl and YEARCOL can be a column name
 or the index of a column. Note that any timezone information is ignored and UTC
@@ -854,40 +847,9 @@ datetime. The lowest start date and highest end date are used to set the
 time period and are returned in the results dictionary in keys startdate
 and enddate.
 
-    dataset.generate_resource_from_iterator(HEADERS, ITERATOR, HXLTAGS,
-    "FOLDER", "FILENAME", RESOURCE_DATA, DATECOL or YEARCOL or DATE_FUNCTION,
-    QUICKCHARTS, "ENCODING")
-
-If desired, `generate_resource_from_iterator` can generate a separate
-QuickCharts resource designed to be used in a time series QuickCharts bite
-provided that the input has #indicator+code, #date and #indicator+value+num.
-This is achieved by supplying the parameter QUICKCHARTS which activates various
-QuickCharts related actions depending upon the keys given in the dictionary.
-The returned dictionary will contain the QuickCharts resource in the key
-qc_resource. If the keys: hashtag - the HXL hashtag to examine - and values -
-the 3 values to look for in that column - are supplied, then a list of booleans
-indicating which QuickCharts bites should be enabled will be returned in the
-key bites_disabled in the returned dictionary. For the 3 values, if the key:
-numeric_hashtag is supplied then if that column for a given value contains no
-numbers, then the corresponding bite will be disabled. If the key: cutdown is
-given, if it is 1, then a separate cut down list is created containing only
-columns with HXL hashtags and rows with desired values (if hashtag and values
-are supplied) for the purpose of driving QuickCharts. It is returned in the key
-qcrows in the returned dictionary with the matching headers in qcheaders. If
-cutdown is 2, then a resource is created using the cut down list. If the key
-cutdownhashtags is supplied, then only the provided hashtags are used for
-cutting down otherwise the full list of HXL tags is used.
-
-The QuickCharts resource will be of form similar to below:
-
-    GHO (CODE),ENDYEAR,Numeric
-    #indicator+code,#date+year+end,#indicator+value+num
-    VIOLENCE_HOMICIDERATE,1994,123.4
-    MDG_0000000001,2015,123.4
-
-`download_and_generate_resource` builds on `generate_resource_from_iterator`.
-It uses an DOWNLOADER, an object of class `Download`, `Retrieve` or other class
-that implements `BaseDownload` to download from URL. Additional arguments in
+`download_generate_resource` builds on `generate_resource`.
+It uses a DOWNLOADER, an object of class `Download`, `Retrieve` or other class
+that implements `BaseDownload` to download from a URL. Additional arguments in
 **KWARGS are passed to the `get_tabular_rows` method of the DOWNLOADER.
 
 Optionally, headers can be inserted at specific positions. This is achieved
@@ -897,12 +859,11 @@ row. If supplied, it takes as arguments: headers (prior to any insertions) and
 row (which will be in dict or list form depending upon the dict_rows argument)
 and outputs a modified row.
 
-The rest of the arguments are the same as for
-`generate_resource_from_iterator`.
+The rest of the arguments are the same as for `generate_resource`.
 
-    dataset.download_and_generate_resource(DOWNLOADER, "URL", HXLTAGS,
-    "FOLDER", "FILENAME", RESOURCE_DATA, HEADER_INSERTIONS, ROW_FUNCTION,
-    DATECOL or YEARCOL or DATE_FUNCTION, QUICKCHARTS, **KWARGS)
+    dataset.download_generate_resource(DOWNLOADER, "URL", "FOLDER", "FILENAME",
+                                       RESOURCE_DATA, HEADER_INSERTIONS, ROW_FUNCTION,
+                                       DATECOL or YEARCOL or DATE_FUNCTION, **KWARGS)
 
 ### QuickCharts Generation
 

pyproject.toml

@@ -37,8 +37,8 @@ dependencies = [
     "ckanapi>=4.8",
     "defopt>=7.0.0",
     "email_validator",
-    "hdx-python-country>=3.9.6",
-    "hdx-python-utilities>=3.9.1",
+    "hdx-python-country>=3.9.8",
+    "hdx-python-utilities>=3.9.5",
     "libhxl>=5.2.2",
     "makefun",
     "quantulum3",

requirements.txt

@@ -4,7 +4,7 @@ annotated-types==0.7.0
     # via pydantic
 astdoc==1.3.2
     # via mkapi
-attrs==25.3.0
+attrs==25.4.0
     # via
     #   frictionless
     #   jsonlines
@@ -14,9 +14,9 @@ babel==2.17.0
     # via mkdocs-material
 backrefs==5.9
     # via mkdocs-material
-cachetools==5.5.2
+cachetools==6.2.1
     # via google-auth
-certifi==2025.8.3
+certifi==2025.10.5
     # via requests
 cfgv==3.4.0
     # via pre-commit
@@ -26,60 +26,59 @@ charset-normalizer==3.4.3
     # via requests
 ckanapi==4.8
     # via hdx-python-api (pyproject.toml)
-click==8.2.1
+click==8.3.0
     # via
     #   mkdocs
-    #   mkdocs-material
     #   typer
 colorama==0.4.6
     # via mkdocs-material
-coverage==7.10.6
+coverage==7.10.7
     # via pytest-cov
 defopt==7.0.0
     # via hdx-python-api (pyproject.toml)
 distlib==0.4.0
     # via virtualenv
-dnspython==2.7.0
+dnspython==2.8.0
     # via email-validator
 docopt==0.6.2
     # via
     #   ckanapi
     #   num2words
-docutils==0.22
+docutils==0.22.2
     # via defopt
 email-validator==2.3.0
     # via hdx-python-api (pyproject.toml)
 et-xmlfile==2.0.0
     # via openpyxl
-filelock==3.19.1
+filelock==3.20.0
     # via virtualenv
 frictionless==5.18.1
     # via hdx-python-utilities
 ghp-import==2.1.0
     # via mkdocs
-google-auth==2.40.3
+google-auth==2.41.1
     # via
     #   google-auth-oauthlib
     #   gspread
 google-auth-oauthlib==1.2.2
     # via gspread
 gspread==6.2.1
     # via hdx-python-api (pyproject.toml)
-hdx-python-country==3.9.6
+hdx-python-country==3.9.8
     # via hdx-python-api (pyproject.toml)
-hdx-python-utilities==3.9.1
+hdx-python-utilities==3.9.5
     # via
     #   hdx-python-api (pyproject.toml)
     #   hdx-python-country
 humanize==4.13.0
     # via frictionless
-identify==2.6.13
+identify==2.6.15
     # via pre-commit
-idna==3.10
+idna==3.11
     # via
     #   email-validator
     #   requests
-ijson==3.4.0
+ijson==3.4.0.post0
     # via hdx-python-utilities
 inflect==7.5.0
     # via quantulum3
@@ -101,7 +100,7 @@ jsonschema==4.25.1
     # via
     #   frictionless
     #   tableschema-to-template
-jsonschema-specifications==2025.4.1
+jsonschema-specifications==2025.9.1
     # via jsonschema
 libhxl==5.2.2
     # via
@@ -111,7 +110,7 @@ loguru==0.7.3
     # via hdx-python-utilities
 makefun==1.16.0
     # via hdx-python-api (pyproject.toml)
-markdown==3.8.2
+markdown==3.9
     # via
     #   mkdocs
     #   mkdocs-material
@@ -120,7 +119,7 @@ markdown-it-py==4.0.0
     # via rich
 marko==2.2.0
     # via frictionless
-markupsafe==3.0.2
+markupsafe==3.0.3
     # via
     #   jinja2
     #   mkdocs
@@ -130,15 +129,15 @@ mergedeep==1.3.4
     # via
     #   mkdocs
     #   mkdocs-get-deps
-mkapi==4.4.5
+mkapi==4.5.0
     # via hdx-python-api (pyproject.toml)
 mkdocs==1.6.1
     # via
     #   mkapi
     #   mkdocs-material
 mkdocs-get-deps==0.2.0
     # via mkdocs
-mkdocs-material==9.6.18
+mkdocs-material==9.6.21
     # via mkapi
 mkdocs-material-extensions==1.3.1
     # via mkdocs-material
@@ -162,7 +161,7 @@ pathspec==0.12.1
     # via mkdocs
 petl==1.7.17
     # via frictionless
-platformdirs==4.4.0
+platformdirs==4.5.0
     # via
     #   mkdocs-get-deps
     #   virtualenv
@@ -184,9 +183,9 @@ pyasn1==0.6.1
     #   rsa
 pyasn1-modules==0.4.2
     # via google-auth
-pydantic==2.11.7
+pydantic==2.12.0
     # via frictionless
-pydantic-core==2.33.2
+pydantic-core==2.41.1
     # via pydantic
 pygments==2.19.2
     # via
@@ -197,14 +196,14 @@ pymdown-extensions==10.16.1
     # via mkdocs-material
 pyphonetics==0.5.3
     # via hdx-python-utilities
-pytest==8.4.1
+pytest==8.4.2
     # via
     #   hdx-python-api (pyproject.toml)
     #   pytest-check
     #   pytest-cov
-pytest-check==2.5.3
+pytest-check==2.6.0
     # via hdx-python-api (pyproject.toml)
-pytest-cov==6.2.1
+pytest-cov==7.0.0
     # via hdx-python-api (pyproject.toml)
 python-dateutil==2.9.0.post0
     # via
@@ -218,7 +217,7 @@ python-slugify==8.0.4
     # via
     #   ckanapi
     #   frictionless
-pyyaml==6.0.2
+pyyaml==6.0.3
     # via
     #   frictionless
     #   mkdocs
@@ -252,7 +251,7 @@ requests-oauthlib==2.0.0
     # via google-auth-oauthlib
 rfc3986==2.0.0
     # via frictionless
-rich==14.1.0
+rich==14.2.0
     # via typer
 rpds-py==0.27.1
     # via
@@ -262,15 +261,15 @@ rsa==4.9.1
     # via google-auth
 ruamel-yaml==0.18.15
     # via hdx-python-utilities
-ruamel-yaml-clib==0.2.12
+ruamel-yaml-clib==0.2.14
     # via ruamel-yaml
 setuptools==80.9.0
     # via ckanapi
 shellingham==1.5.4
     # via typer
 simpleeval==1.0.3
     # via frictionless
-simplejson==3.20.1
+simplejson==3.20.2
     # via ckanapi
 six==1.17.0
     # via
@@ -292,7 +291,7 @@ text-unidecode==1.3
     # via python-slugify
 typeguard==4.4.4
     # via inflect
-typer==0.17.3
+typer==0.19.2
     # via frictionless
 typing-extensions==4.15.0
     # via
@@ -302,7 +301,7 @@ typing-extensions==4.15.0
     #   typeguard
     #   typer
     #   typing-inspection
-typing-inspection==0.4.1
+typing-inspection==0.4.2
     # via pydantic
 unidecode==1.4.0
     # via
@@ -314,7 +313,7 @@ urllib3==2.5.0
     #   requests
 validators==0.35.0
     # via frictionless
-virtualenv==20.34.0
+virtualenv==20.35.3
     # via pre-commit
 watchdog==6.0.0
     # via mkdocs
@@ -326,7 +325,7 @@ xlrd3==1.1.0
     # via libhxl
 xlsx2csv==0.8.4
     # via hdx-python-utilities
-xlsxwriter==3.2.5
+xlsxwriter==3.2.9
     # via tableschema-to-template
 xlwt==1.3.0
     # via hdx-python-utilities