roachtest: django failed [release-25.3: disallowed due to schema_locked]

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.django failed with artifacts on release-25.3 @ 35b2d15446aa78530a4ff85f359764cb35d033d0:

--- FAIL: transactions.tests.AtomicErrorsTests.test_atomic_prevents_queries_in_broken_transaction_after_client_close (unexpected)
--- FAIL: backends.tests.ThreadTests.test_pass_connection_between_threads (unexpected)
--- FAIL: transactions.tests.AtomicTests.test_rollback (unexpected)
--- FAIL: transactions.tests.AtomicInsideTransactionTests.test_merged_commit_rollback (unexpected)
--- FAIL: transactions.tests.AtomicWithoutAutocommitTests.test_merged_rollback_commit (unexpected)
--- FAIL: transactions.tests.AtomicTests.test_reuse_commit_commit (unexpected)
--- FAIL: transactions.tests.DurableTransactionTests.test_nested_outer_durable (unexpected)
--- FAIL: select_for_update.tests.SelectForUpdateTests.test_select_for_update_with_get (unexpected)
--- FAIL: transactions.tests.AtomicInsideTransactionTests.test_decorator_syntax_rollback (unexpected)
--- FAIL: transaction_hooks.tests.TestConnectionOnCommit.test_no_hooks_run_from_failed_transaction (unexpected)
--- FAIL: transaction_hooks.tests.TestConnectionOnCommit.test_discards_hooks_from_rolled_back_savepoint (unexpected)
--- FAIL: transaction_hooks.tests.TestConnectionOnCommit.test_does_not_execute_if_transaction_rolled_back (unexpected)
--- FAIL: transactions.tests.AtomicTests.test_merged_commit_rollback (unexpected)
--- FAIL: transactions.tests.AtomicWithoutAutocommitTests.test_rollback (unexpected)
--- FAIL: auth_tests.test_migrations.ProxyModelWithSameAppLabelTests.test_user_keeps_same_permissions_after_migrating_backward (unexpected)
--- FAIL: transactions.tests.AtomicTests.test_reuse_rollback_rollback (unexpected)
--- FAIL: transaction_hooks.tests.TestConnectionOnCommit.test_runs_hooks_in_order_registered (unexpected)
--- FAIL: transactions.tests.AtomicTests.test_alternate_decorator_syntax_rollback (unexpected)
--- FAIL: transactions.tests.AtomicMergeTests.test_merged_inner_savepoint_rollback (unexpected)
--- FAIL: transactions.tests.AtomicWithoutAutocommitTests.test_force_rollback (unexpected)
--- FAIL: transactions.tests.AtomicWithoutAutocommitTests.test_nested_rollback_commit (unexpected)
--- FAIL: auth_tests.test_handlers.ModWsgiHandlerTestCase.test_groups_for_user (unexpected)
--- FAIL: transactions.tests.AtomicWithoutAutocommitTests.test_alternate_decorator_syntax_rollback (unexpected)
--- FAIL: transaction_hooks.tests.TestConnectionOnCommit.test_executes_immediately_if_no_transaction (unexpected)
--- FAIL: transactions.tests.AtomicWithoutAutocommitTests.test_decorator_syntax_rollback (unexpected)
--- FAIL: transaction_hooks.tests.TestConnectionOnCommit.test_delays_execution_until_after_transaction_commit (unexpected)
--- FAIL: transactions.tests.AtomicWithoutAutocommitTests.test_prevent_rollback (unexpected)
--- FAIL: select_for_update.tests.SelectForUpdateTests.test_for_update_of_followed_by_values_list (unexpected)
--- FAIL: transactions.tests.AtomicInsideTransactionTests.test_nested_rollback_rollback (unexpected)
--- FAIL: transactions.tests.AtomicTests.test_alternate_decorator_syntax_commit (unexpected)
--- FAIL: transactions.tests.AtomicWithoutAutocommitTests.test_merged_commit_rollback (unexpected)
--- FAIL: transactions.tests.AtomicInsideTransactionTests.test_merged_rollback_commit (unexpected)
--- FAIL: backends.tests.BackendTestCase.test_cursor_executemany_with_pyformat (unexpected)
--- FAIL: transaction_hooks.tests.TestConnectionOnCommit.test_transaction_in_hook (unexpected)
--- FAIL: backends.tests.LongNameTest.test_sequence_name_length_limits_flush (unexpected)
--- FAIL: servers.tests.LiveServerTestCloseConnectionTest.test_closes_connections (unexpected)
--- FAIL: transactions.tests.AtomicTests.test_reuse_commit_rollback (unexpected)
--- FAIL: transaction_hooks.tests.TestConnectionOnCommit.test_hooks_cleared_on_reconnect (unexpected)
--- FAIL: contenttypes_tests.test_operations.ContentTypeOperationsTests.test_existing_content_type_rename (unexpected)
--- FAIL: transactions.tests.AtomicInsideTransactionTests.test_nested_rollback_commit (unexpected)
--- FAIL: transactions.tests.AtomicTests.test_merged_rollback_rollback (unexpected)
--- FAIL: transactions.tests.AtomicInsideTransactionTests.test_reuse_rollback_rollback (unexpected)
--- FAIL: transaction_hooks.tests.TestConnectionOnCommit.test_hooks_cleared_after_rollback (unexpected)
--- FAIL: transactions.tests.AtomicTests.test_nested_commit_rollback (unexpected)
--- FAIL: transactions.tests.AtomicWithoutAutocommitTests.test_reuse_rollback_rollback (unexpected)
--- FAIL: transactions.tests.AtomicTests.test_reuse_rollback_commit (unexpected)
--- FAIL: backends.base.test_operations.SqlFlushTests.test_execute_sql_flush_statements (unexpected)
For a full summary look at the django artifacts 
An updated blocklist (djangoBlocklist) is available in the artifacts' django log
test artifacts and logs in: /artifacts/django/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=16
• encrypted=false
• fs=ext4
• localSSD=true
• metamorphicLeases=leader
• metamorphicWriteBuffering=true
• runtimeAssertionsBuild=true
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: django failed [release-25.3: disallowed due to schema_locked] #154610 roachtest: django failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-sql-foundations branch-release-25.3.3-rc release-blocker]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-55045

[rafiss]

======================================================================
ERROR: test_select_for_update_with_get (select_for_update.tests.SelectForUpdateTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/data1/django/django/db/backends/utils.py", line 87, in _execute
    return self.cursor.execute(sql)
psycopg2.errors.OperatorIntervention: this schema change is disallowed because table "select_for_update_eucountry" is locked and this operation cannot automatically unlock the table
DETAIL:  To unlock the table, execute `ALTER TABLE select_for_update_eucountry SET (schema_locked = false);`
After the schema change completes, we recommend setting it back to true with `ALTER TABLE select_for_update_eucountry SET (schema_locked = true);`.
HINT:  Locking the table improves changefeed performance; see https://www.cockroachlabs.com/docs/v25.3/changefeed-best-practices.html#lock-the-schema-on-changefeed-watched-tables


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/data1/django/django/core/management/commands/flush.py", line 73, in handle
    connection.ops.execute_sql_flush(sql_list)
  File "/home/ubuntu/venv/lib/python3.10/site-packages/django_cockroachdb/operations.py", line 90, in execute_sql_flush
    return super().execute_sql_flush(sql_list)
  File "/mnt/data1/django/django/db/backends/base/operations.py", line 450, in execute_sql_flush
    cursor.execute(sql)
  File "/mnt/data1/django/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/mnt/data1/django/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/mnt/data1/django/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/mnt/data1/django/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/mnt/data1/django/django/db/backends/utils.py", line 87, in _execute
    return self.cursor.execute(sql)
django.db.utils.OperationalError: this schema change is disallowed because table "select_for_update_eucountry" is locked and this operation cannot automatically unlock the table
DETAIL:  To unlock the table, execute `ALTER TABLE select_for_update_eucountry SET (schema_locked = false);`
After the schema change completes, we recommend setting it back to true with `ALTER TABLE select_for_update_eucountry SET (schema_locked = true);`.
HINT:  Locking the table improves changefeed performance; see https://www.cockroachlabs.com/docs/v25.3/changefeed-best-practices.html#lock-the-schema-on-changefeed-watched-tables

seems like #154530 didn't fix this