fix: handle inf/-inf in ShimSparkErrorConverter cast overflow#3768
Open
manuzhang wants to merge 1 commit intoapache:mainfrom
Open
fix: handle inf/-inf in ShimSparkErrorConverter cast overflow#3768manuzhang wants to merge 1 commit intoapache:mainfrom
manuzhang wants to merge 1 commit intoapache:mainfrom
Conversation
Member
Author
|
@parthchandra Please take a look when you find time. |
manuzhang
force-pushed
the
fix-shim-cast-overflow-inf
branch
from
ad224ca to
f38fd91
Compare
andygrove
reviewed
Mar 24, 2026
|
|
||
| private def parseFloatLiteral(value: String): Float = { | ||
| value.toLowerCase match { | ||
| case "inf" | "+inf" | "infinity" | "+infinity" => Float.PositiveInfinity |
Member
There was a problem hiding this comment.
I know this issue is focused on inf but do we need to do anything with nan as well?
Normalize inf/nan literals for float/double cast overflow conversion across Spark 3.4/3.5/4.0 and add unit tests in SparkErrorConverterSuite for float/double inf/-inf/nan. Co-authored-by: Codex <[email protected]>
manuzhang
force-pushed
the
fix-shim-cast-overflow-inf
branch
from
f38fd91 to
5068336
Compare
| } | ||
|
|
||
| private def parseDoubleLiteral(value: String): Double = { | ||
| value.toLowerCase match { |
Contributor
There was a problem hiding this comment.
In conversion_funcs/numeric.rs:spark_cast_nonintegral_numeric_to_integral the calls to cast_float_to_int16_down and cast_float_to_int32_up explicitly format the string with "{:e}D" (a suffix D).
I think inf and nan will get this D suffix and the resultant string infD or nanD would not match.
The unit tests below will not catch this either.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #3767.
Rationale for this change
Fixes incorrect exception translation for overflow cases involving infinity literals and aligns Comet behavior with Spark expectations in ANSI mode.
What changes are included in this PR?
Normalize inf literals for float/double cast overflow conversion across Spark 3.4/3.5/4.0 and add unit tests in SparkErrorConverterSuite.
How are these changes tested?
Add new UT SparkErrorConverterSuite.