SNOW-2327397: Schema is not inferred during write operations

[andrea-santoro1-a2a]

1. What version of Python are you using?

   3.12

2. What operating system and processor architecture are you using?

   Docker image python:3.12-slim

3. What are the component versions in the environment (pip freeze)?

   snowflake-snowpark-python==1.38.0

4. What did you do?

   When performing write operations, such as uploading a dataframe to a table or performing a merge over an existing table, schema is not inferred from the given table name even if the table name has a (semi-)fully qualified name (for example schema.table or db.schema.table).

   The resulting error, however states that using the (semi-)fully qualified name is indeed an option. For example, this is the error message shown when performing a merge with fully qualified table names:

   ClientError: AlgorithmError: SnowparkSQLException("090106 (22000): 01beeed7-0305-2a87-0001-8d0a1f96a5d6: Cannot perform DROP. This session does not have a current schema. Call 'USE SCHEMA', or use a qualified name.", '1304', '01beeed7-0305-2a87-0001-8d0a1f96a5d6'), exit code: 1

   By explicitly setting the schema with sql (session.sql("USE SCHEMA ...").collect()), the same merge operation succeeds.

5. What did you expect to see?

   Infer and automatically set the right schema when table name is formatted as schema.table or db.schema.table, as the above error suggests

6. Can you set logging to DEBUG and collect the logs?

   I'll add this if needed, doesn't seem useful to me

[sfc-gh-aling]

hey @andrea-santoro1-a2a , thanks for reaching out!
could you share a code example to help me reproduce the issue?

[andrea-santoro1-a2a]

Sure, both mentioned issues below:

from snowflake.snowpark import Session

connection_params = {
  'user': sf_user,
  'private_key': pkey,
  'account': account,
  'warehouse': SF_WAREHOUSE,
  'database': SF_PROJECT_ID
  # here you can see i'm not setting a schema
}

session = Session.builder.configs(connection_params).create()

# merge issue
existing_table = session.table('schema.existing_table')
df = session.create_dataframe(
  ...,
  session.table('schema.another_existing_table')\
    .select(...)\ # at least one join key must be present
    .schema
)

docs_embeddings_merge_result = existing_table.merge(
  df,
  join_expr=merge_condition,
  clauses=[
      ...
  ]
)

# write into table issue
session\
  .read.parquet('@existing_stage')\
  .write.mode('truncate').save_as_table(
      "schema.yet_another_existing_table" # same with "db.schema.yet_another_existing_table"
  )

[austb-cloudbolt]

This also appears to apply to databases as well. Setting USE DATABASE allowed us to get past

snowflake.snowpark.exceptions.SnowparkSQLException: (1304): 01bf3a1d-0206-8142-0003-896f06ca573e: 090105 (22000): 01bf3a1d-0206-8142-0003-896f06ca573e: Cannot perform DROP. This session does not have a current database. Call 'USE DATABASE', or use a qualified name.
   in function ..... with handler main
  
  This error sometimes occurs when invalid credentials are provided, or when your default role does not have access to use the specified database. Please double check your profile and try again.

Our error also appears to be size related, and is only happening in the production environment. I haven't definitively tied it to the large_query_breakdown.py code but that seems to line up with what we are seeing.