Skip to content

Conversation

@NickCrews
Copy link
Contributor

This is a sanity check. If sqlglot tells us we are generating invalid SQL, then we should be handling it better.

I'm not sure how CI will like this. Potentially this will reveal a lot of bad assumptions we have been making.

My other motivation here is that if we merge this, then we can push more logic down into SQLglot. For example, if tobymao/sqlglot#6377 gets merged, then we can rely on sqlglot to tell us that column.first(include_nulls=False) is illegal to compile on postgres (only column.first(include_nulls=True) is possible to correctly compile. Currently, we have to do these sorts of checks in our individual backend compilers, for example see #11311

@cpcloud
Copy link
Member

cpcloud commented Nov 20, 2025

Wow, I am surprised that only 2 backends failed with this enabled!

@NickCrews
Copy link
Contributor Author

@cpcloud if I fix those failing tests then does this seem like a good idea?

@NickCrews NickCrews force-pushed the error-on-sqlglot-unsupported branch from 76a4475 to 2848be3 Compare December 22, 2025 18:36
NickCrews added a commit to NickCrews/sqlglot that referenced this pull request Dec 22, 2025
This is for `SELECT * FROM t1 INTERSECT/EXCEPT ALL SELECT * FROM t2`
Trino inherits the dialect from Presto. Presto doesn't support the `ALL` modifier, but trino does, see trinodb/trino#5890

We ran into this in ibis, where we are indeed able to compile and successfully execute the ALL modifier, but as soon as [we turned on sqlglot errors](ibis-project/ibis#11772), we then started getting spurious sqlglot errors: https://github.com/ibis-project/ibis/actions/runs/19549750869/job/55977864377?pr=11772
@NickCrews
Copy link
Contributor Author

I think I fixed the flink error, that was easy.

For trino, I started tobymao/sqlglot#6616 to fix one of the trino errors. I still need to work on the other trino error.

@github-actions github-actions bot added the datatypes Issues relating to ibis's datatypes (under `ibis.expr.datatypes`) label Dec 22, 2025
VaggelisD pushed a commit to tobymao/sqlglot that referenced this pull request Dec 23, 2025
This is for `SELECT * FROM t1 INTERSECT/EXCEPT ALL SELECT * FROM t2`
Trino inherits the dialect from Presto. Presto doesn't support the `ALL` modifier, but trino does, see trinodb/trino#5890

We ran into this in ibis, where we are indeed able to compile and successfully execute the ALL modifier, but as soon as [we turned on sqlglot errors](ibis-project/ibis#11772), we then started getting spurious sqlglot errors: https://github.com/ibis-project/ibis/actions/runs/19549750869/job/55977864377?pr=11772
georgesittas pushed a commit to tobymao/sqlglot that referenced this pull request Dec 24, 2025
This is for `SELECT * FROM t1 INTERSECT/EXCEPT ALL SELECT * FROM t2`
Trino inherits the dialect from Presto. Presto doesn't support the `ALL` modifier, but trino does, see trinodb/trino#5890

We ran into this in ibis, where we are indeed able to compile and successfully execute the ALL modifier, but as soon as [we turned on sqlglot errors](ibis-project/ibis#11772), we then started getting spurious sqlglot errors: https://github.com/ibis-project/ibis/actions/runs/19549750869/job/55977864377?pr=11772
@NickCrews NickCrews force-pushed the error-on-sqlglot-unsupported branch from 2848be3 to a0f0f1a Compare January 26, 2026 05:30
@NickCrews NickCrews force-pushed the error-on-sqlglot-unsupported branch from a0f0f1a to f75a64e Compare January 27, 2026 22:55
@NickCrews
Copy link
Contributor Author

NickCrews commented Jan 27, 2026

Repro for the trino error for playing around:

import ibis
from ibis.backends.sql.compilers.trino import TrinoCompiler
import sqlglot as sg

data = {"a": [[2], [4]]}

t = ibis.memtable(data, schema=dict(a="!array<int8>"))

predicate = lambda x, i: x + (i - i) > 1
# predicate = partial(lambda x, y, i: x > y + (i * 0), y=1)
filtered = t.a.filter(predicate)

sge = TrinoCompiler().to_sqlglot(filtered)
sge.sql(
    dialect="trino",
    pretty=True,
    copy=False,
    unsupported_level=sg.ErrorLevel.RAISE,
)

Edit: sqlglot warns when compiling a sge.Struct on trino/presto if the type of any of the fields can't be inferred

We trigger this when compiling ops.ArrayFilter on trino, and in the temp struct we go through, we just pass a bare identifier expression, which sqlglot can't infer the type of.

So the solution was to pass a type hint to sqlglot the type of this struct field. Maybe not ideal sqlglot behavior, I feel like it should be valid to compile an untyped field with no warning, but we can work around it.

@NickCrews NickCrews force-pushed the error-on-sqlglot-unsupported branch from 8a0adf7 to e780287 Compare January 28, 2026 01:03
@NickCrews
Copy link
Contributor Author

OK @cpcloud all tests pass now!

One possible thing to think about here: There may be cases that work just fine, but still raise sqlglot errors, that we aren't seeing in tests. If we all of a sudden make these error, then we might just be breaking some poor user for no real reason. So, perhaps we want to only enable this strictness check in tests? eg by looking at some environment variable that is set in conftest.py?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datatypes Issues relating to ibis's datatypes (under `ibis.expr.datatypes`) sql Backends that generate SQL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants