Deprecations, documentation and clean up in preparation for 5.0

NEWS.md

@@ -3,6 +3,8 @@ MixedModels v5.0.0 Release Notes
 - Optimization is now performed _without constraints_. In a post-fitting step, the Cholesky factor is canonicalized to have non-negative diagonal elements. [#840]
 - The default optimizer has changed to NLopt's implementation of NEWUOA where possible. NLopt's implementation fails on 1-dimensional problems, so in the case of a single, scalar random effect, BOBYQA is used instead. In the future, the default optimizer backend will likely change to PRIMA and NLopt support will be moved to an extension. Blocking this change in backend is an issue with PRIMA.jl when running in VSCode's built-in REPL on Linux. [#840]
 - [BREAKING] Support for constrained optimization has been completely removed, i.e. the field `lowerbd` has been removed from `OptSummary`. [#849]
+- [BREAKING] The deprecated `use_threads` kwarg has been dropped from `parametricbootstrap`. It had been a no-op since v4.10.0. [#841]
+- [BREAKING] The deprecated `hide_progress` kwarg has been dropped from `parametricbootstrap`. It had been replaced by `progress` since v4.22.0. [#841]
 - [BREAKING] A fitlog is always kept -- the deprecated keyword argument `thin` has been removed as has the `fitlog` keyword argument. [#850]
 - The fitlog is now stored as Tables.jl-compatible column table. [#850]
 - Internal code around the default optimizer has been restructured. In particular, the NLopt backend has been moved to a submodule, which will make it easier to move it to an extension if we promote another backend to the default. [#853]
@@ -13,6 +15,7 @@ MixedModels v5.0.0 Release Notes
 - One argument `predict(::GeneralizedLinearMixedModel)`, i.e. prediction on the original data, now supports the `type` keyword argument. [#856]
 - `isnested(A::ReMat, B::ReMat)` is now a method of `StatsModels.isnested`.[#858]
 - [BREAKING ]`likelihoodratiotest` has been reworked to be a thin wrapper around `StatsModels.lrtest`. The historical difference in behavior in terms of nesting checks created some confusion. Users advanced enough to create models with non-obvious nesting are assumed to be advanced enough to manually compute the likelihood ratio test. The function `likelihoodratiotest` and associated `LikelihoodRatioTest` type (now with a type parameter for number of models) has been kept to enable printing of test results with model formulae. Most users should not notice a difference in behavior, but the display has been slightly changed and the internal field structure has changed.[#858]
+- Failures to fit a spline in profiling now generates a more helpful error message. [#857]
 
 MixedModels v4.38.0 Release Notes
 ==============================
@@ -680,4 +683,5 @@ Package dependencies
 [#854]: https://github.com/JuliaStats/MixedModels.jl/issues/854
 [#855]: https://github.com/JuliaStats/MixedModels.jl/issues/855
 [#856]: https://github.com/JuliaStats/MixedModels.jl/issues/856
+[#857]: https://github.com/JuliaStats/MixedModels.jl/issues/857
 [#858]: https://github.com/JuliaStats/MixedModels.jl/issues/858

Project.toml

@@ -1,7 +1,7 @@
 name = "MixedModels"
 uuid = "ff71e718-51f3-5ec2-a782-8ffcbfa3c316"
 author = ["Phillip Alday <me@phillipalday.com>", "Douglas Bates <dmbates@gmail.com>"]
-version = "5.0.0-DEV"
+version = "5.0.0"
 
 [deps]
 Arrow = "69666777-d1a9-59fb-9406-91d4454c9d45"

README.md

@@ -53,31 +53,33 @@ Typical distribution forms are _Bernoulli_ for binary data or _Poisson_ for coun
 |OS      | OS Version    |Arch    |Julia           |
 |:------:|:-------------:|:------:|:--------------:|
 |Linux   | Ubuntu 22.04  | x64    |v1.10           |
-|Linux   | Ubuntu 22.04  | x64    |current release |
+|Linux   | Ubuntu 24.04  | x64    |current release |
 |Linux   | Ubuntu 22.04  | x64    |nightly         |
 |macOS   | Sonoma 14     | aarm64 |v1.10           |
+|macOS   | Sequoia 15    | aarm64 |current release |
 |Windows | Server 2022   | x64    |v1.10           |
 
 Note that previous releases still support older Julia versions.
 
-## Version 4.0.0
+## Version 5.0
 
-Version 4.0.0 contains some user-visible changes and many changes in the underlying code.
+Version 5.0.0 contains some user-visible changes and many changes in the underlying code.
 
 Please see [NEWS](NEWS.md) for a complete overview, but a few key points are:
-
-- The internal storage of the model matrices in `LinearMixedModel` has changed and been optimized. This change should be transparent to users who are not manipulating the fields of the model `struct` directly.
-- The [handling of rank deficiency](https://juliastats.org/MixedModels.jl/v4.0/rankdeficiency/) continues to evolve.
-- Additional [`predict` and `simulate`](https://juliastats.org/MixedModels.jl/v4.0/prediction/) methods have been added for generalizing to new data.
-- `saveoptsum` and `restoreoptsum!` provide for saving and restoring the `optsum` and thus offer a way to serialize a model fit.
-- There is improved support for the runtime construction of model formula, especially `RandomEffectsTerm`s and nested terms (methods for `Base.|(::AbstractTerm, ::AbstractTerm)` and `Base./(::AbstractTerm, ::AbstractTerm)`).
-- A progress display is shown by default for models taking more than a few hundred milliseconds to fit. This can be disabled with the keyword argument `progress=false`.
+- Options related to multithreading in the bootstrap have been completely removed.
+- Model fitting now uses unconstrained optimization, with a post-fit canonicalization step so that the diagonal elements of the lower Cholesky factor are non-negative. Relatedly, support for constrained optimization has been completely removed and the `lowerbd` field of `OptSummary` dropped.
+- The default optimizer has changed to use NLopt's implementation of NEWUOA. Further changes to the default optimizer are considered non-breaking.
+- The `profile` function now respects backend and optimizer settings.
+- The deprecated `hide_progress` keyword argument has been removed in favor of the shorter and affirmative `progress`.
+- A fitlog is always kept and stored as a Tables.jl-compatible column table.
 
 ## Quick Start
 ```julia-repl
 julia> using MixedModels
 
-julia> m1 = fit(MixedModel, @formula(yield ~ 1 + (1|batch)), MixedModels.dataset(:dyestuff))
+julia> using MixedModelsDatasets: dataset
+
+julia> m1 = lmm(@formula(yield ~ 1 + (1|batch)), dataset(:dyestuff))
 Linear mixed model fit by maximum likelihood
  yield ~ 1 + (1 | batch)
    logLik   -2 logLik     AIC       AICc        BIC
@@ -86,7 +88,7 @@ Linear mixed model fit by maximum likelihood
 Variance components:
             Column    Variance Std.Dev.
 batch    (Intercept)  1388.3332 37.2603
-Residual              2451.2501 49.5101
+Residual              2451.2500 49.5101
  Number of obs: 30; levels of grouping factors: 6
 
   Fixed-effects parameters:
@@ -98,62 +100,31 @@ Residual              2451.2501 49.5101
 
 julia> using Random
 
-julia> bs = parametricbootstrap(MersenneTwister(42), 1000, m1);
+julia> bs = parametricbootstrap(MersenneTwister(42), 1000, m1)
 Progress: 100%%|████████████████████████████████████████████████| Time: 0:00:00
-
-julia> propertynames(bs)
-13-element Vector{Symbol}:
- :allpars
- :objective
- :σ
- :β
- :se
- :coefpvalues
- :θ
- :σs
- :λ
- :inds
- :lowerbd
- :fits
- :fcnames
+MixedModelBootstrap with 1000 samples
+     parameter  min      q25       median    mean      q75       max
+   ┌────────────────────────────────────────────────────────────────────
+ 1 │ β1         1474.0   1515.62   1527.68   1527.4    1539.56   1584.57
+ 2 │ σ          26.6353  43.7165   48.4817   48.8499   53.8964   73.8684
+ 3 │ σ1         0.0      16.835    28.1067   27.7039   39.491    83.688
+ 4 │ θ1         0.0      0.340364  0.561701  0.588678  0.840284  2.24396
 
 julia> bs.coefpvalues # returns a row table
-1000-element Vector{NamedTuple{(:iter, :coefname, :β, :se, :z, :p), Tuple{Int64, Symbol, Float64, Float64, Float64, Float64}}}:
- (iter = 1, coefname = Symbol("(Intercept)"), β = 1517.0670832927115, se = 20.76271142094811, z = 73.0669059804057, p = 0.0)
- (iter = 2, coefname = Symbol("(Intercept)"), β = 1503.5781855888436, se = 8.1387737362628, z = 184.7425956676446, p = 0.0)
- (iter = 3, coefname = Symbol("(Intercept)"), β = 1529.2236379016574, se = 16.523824785737837, z = 92.54659001356465, p = 0.0)
- ⋮
- (iter = 998, coefname = Symbol("(Intercept)"), β = 1498.3795009457242, se = 25.649682012258104, z = 58.417079019913054, p = 0.0)
- (iter = 999, coefname = Symbol("(Intercept)"), β = 1526.1076747922416, se = 16.22412120273579, z = 94.06411945042063, p = 0.0)
- (iter = 1000, coefname = Symbol("(Intercept)"), β = 1557.7546433870125, se = 12.557577103806015, z = 124.04898098653763, p = 0.0)
 
 julia> using DataFrames
 
 julia> DataFrame(bs.coefpvalues) # puts it into a DataFrame
 1000×6 DataFrame
-│ Row  │ iter  │ coefname    │ β       │ se      │ z       │ p       │
-│      │ Int64 │ Symbol      │ Float64 │ Float64 │ Float64 │ Float64 │
-├──────┼───────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
-│ 1    │ 1     │ (Intercept) │ 1517.07 │ 20.7627 │ 73.0669 │ 0.0     │
-│ 2    │ 2     │ (Intercept) │ 1503.58 │ 8.13877 │ 184.743 │ 0.0     │
-│ 3    │ 3     │ (Intercept) │ 1529.22 │ 16.5238 │ 92.5466 │ 0.0     │
-⋮
-│ 998  │ 998   │ (Intercept) │ 1498.38 │ 25.6497 │ 58.4171 │ 0.0     │
-│ 999  │ 999   │ (Intercept) │ 1526.11 │ 16.2241 │ 94.0641 │ 0.0     │
-│ 1000 │ 1000  │ (Intercept) │ 1557.75 │ 12.5576 │ 124.049 │ 0.0     │
-
-julia> DataFrame(bs.β)
-1000×3 DataFrame
-│ Row  │ iter  │ coefname    │ β       │
-│      │ Int64 │ Symbol      │ Float64 │
-├──────┼───────┼─────────────┼─────────┤
-│ 1    │ 1     │ (Intercept) │ 1517.07 │
-│ 2    │ 2     │ (Intercept) │ 1503.58 │
-│ 3    │ 3     │ (Intercept) │ 1529.22 │
-⋮
-│ 998  │ 998   │ (Intercept) │ 1498.38 │
-│ 999  │ 999   │ (Intercept) │ 1526.11 │
-│ 1000 │ 1000  │ (Intercept) │ 1557.75 │
+  Row │ iter   coefname     β        se       z         p
+      │ Int64  Symbol       Float64  Float64  Float64   Float64
+──────┼─────────────────────────────────────────────────────────
+    1 │     1  (Intercept)  1552.65   9.8071  158.319       0.0
+    2 │     2  (Intercept)  1557.33  21.0679   73.9197      0.0
+  ⋮   │   ⋮         ⋮          ⋮        ⋮        ⋮         ⋮
+  999 │   999  (Intercept)  1503.1   30.3349   49.5501      0.0
+ 1000 │  1000  (Intercept)  1565.47  24.5067   63.8794      0.0
+                                                996 rows omitted
 ```
 
 ## Funding Acknowledgement

src/bootstrap.jl

@@ -219,18 +219,9 @@ function parametricbootstrap(
     β::AbstractVector=fixef(morig),
     σ=morig.σ,
     θ::AbstractVector=morig.θ,
-    use_threads::Bool=false,
     progress::Bool=true,
-    hide_progress::Union{Bool,Nothing}=nothing,
     optsum_overrides=(;),
 ) where {T}
-    if !isnothing(hide_progress)
-        Base.depwarn(
-            "`hide_progress` is deprecated, please use `progress` instead." *
-            "NB: `progress` is a positive action, i.e. `progress=true` means show the progress bar.",
-            :parametricbootstrap; force=true)
-        progress = !hide_progress
-    end
     if σ !== missing
         σ = T(σ)
     end
@@ -248,10 +239,6 @@ function parametricbootstrap(
 
     β_names = Tuple(Symbol.(coefnames(morig)))
 
-    use_threads && Base.depwarn(
-        "use_threads is deprecated and will be removed in a future release",
-        :parametricbootstrap,
-    )
     samp = replicate(n; progress) do
         simulate!(rng, m; β, σ, θ)
         refit!(m; progress=false)

src/utilities.jl

@@ -129,19 +129,8 @@ Return a vector of the values of `n` calls to `f()` - used in simulations where
 bar is automatically disabled for non-interactive (i.e. logging) contexts.
 """
 function replicate(
-    f::Function, n::Integer; use_threads=false, hide_progress=nothing, progress=true
+    f::Function, n::Integer; progress=true
 )
-    use_threads && Base.depwarn(
-        "use_threads is deprecated and will be removed in a future release",
-        :replicate,
-    )
-    if !isnothing(hide_progress)
-        Base.depwarn(
-            "`hide_progress` is deprecated, please use `progress` instead." *
-            "NB: `progress` is a positive action, i.e. `progress=true` means show the progress bar.",
-            :replicate; force=true)
-        progress = !hide_progress
-    end
     # and we want some advanced options
     p = Progress(n; output=Base.stderr, enabled=progress && !_is_logging(stderr))
     # get the type

test/bootstrap.jl

@@ -16,7 +16,7 @@ include("modelcache.jl")
 
 function quickboot(m, n=2)
     return parametricbootstrap(MersenneTwister(42), n, m;
-        progress=false, use_threads=false,
+        progress=false,
         optsum_overrides=(; ftol_rel=1e-8))
 end
 
@@ -92,11 +92,8 @@ end
     # two implicit tests
     # 1. type conversion of ints to floats
     # 2. test method for default RNG
-    @test_logs((:warn, r"hide_progress"),
-        parametricbootstrap(1, fm, β=[1], σ=1, hide_progress=true))
-
     bsamp = parametricbootstrap(MersenneTwister(1234321), 100, fm;
-        use_threads=false, progress=false)
+        progress=false)
     @test isa(propertynames(bsamp), Vector{Symbol})
     @test length(bsamp.objective) == 100
     @test keys(first(bsamp.fits)) == (:objective, :σ, :β, :se, :θ)
@@ -107,13 +104,13 @@ end
 
     @testset "optsum_overrides" begin
         bsamp2 = parametricbootstrap(MersenneTwister(1234321), 100, fm;
-            use_threads=false, progress=false,
+            progress=false,
             optsum_overrides=(; ftol_rel=1e-8))
         # for such a simple, small model setting the function value
         # tolerance has little effect until we do something extreme
         @test bsamp.objective ≈ bsamp2.objective
         bsamp2 = parametricbootstrap(MersenneTwister(1234321), 100, fm;
-            use_threads=false, progress=false,
+            progress=false,
             optsum_overrides=(; ftol_rel=1.0))
         @test !(bsamp.objective ≈ bsamp2.objective)
     end
@@ -131,12 +128,6 @@ end
     @test only(unique(coefp.coefname)) == Symbol("(Intercept)")
     @test propertynames(coefp) == [:iter, :coefname, :β, :se, :z, :p]
 
-    @testset "threaded bootstrap" begin
-        @test_logs (:warn, r"use_threads is deprecated") parametricbootstrap(
-            MersenneTwister(1234321), 1, fm;
-            use_threads=true, progress=false)
-    end
-
     @testset "zerocorr + Base.length + ftype" begin
         fmzc = models(:sleepstudy)[2]
         pbzc = parametricbootstrap(MersenneTwister(42), 5, fmzc, Float16;

test/utilities.jl

@@ -39,11 +39,6 @@ end
     @test isconstant(Union{Int,Missing}[missing, missing, missing])
 end
 
-@testset "replicate" begin
-    @test_logs (:warn, r"use_threads is deprecated") replicate(string, 1; use_threads=true)
-    @test_logs (:warn, r"hide_progress") replicate(string, 1; hide_progress=true)
-end
-
 @testset "PCA" begin
     io = IOBuffer()
     pca = models(:kb07)[3].PCA.item