Sync repos: Release 170.157.0

.github/instructions/bug_fixing.guidelines.instructions.md

@@ -90,6 +90,27 @@ The process of fixing a bug, especially one that involves adding new syntax, fol
 
 By following these steps, you can ensure that new syntax is correctly parsed, represented in the AST, generated back into a script, and fully validated by the testing framework without breaking existing functionality.
 
+## Testing Best Practices
+
+### ✅ DO: Use Existing Test Framework
+- Add test methods to existing test classes like `Only170SyntaxTests.cs`
+- Use the established `TSqlParser.Parse()` pattern for verification
+- Example:
+  ```csharp
+  [TestMethod]
+  public void VerifyNewSyntax()
+  {
+      var parser = new TSql170Parser(true);
+      var result = parser.Parse(new StringReader("YOUR SQL HERE"), out var errors);
+      Assert.AreEqual(0, errors.Count, "Should parse without errors");
+  }
+  ```
+
+### ❌ DON'T: Create New Test Projects
+- **Never** create standalone `.csproj` files for testing parser functionality
+- **Never** create new console applications or test runners
+- This causes build issues and doesn't integrate with the existing test infrastructure
+
 ## Special Case: Extending Grammar Rules from Literals to Expressions
 
 A common type of bug involves extending existing grammar rules that only accept literal values (like integers or strings) to accept full expressions (parameters, variables, outer references, etc.). This pattern was used to fix the VECTOR_SEARCH TOP_N parameter issue.

.github/instructions/grammer.guidelines.instructions.md

@@ -64,17 +64,32 @@ yourContextParameterRule returns [ScalarExpression vResult]
 Most script generators using `GenerateNameEqualsValue()` or similar methods work automatically with `ScalarExpression`. No changes typically needed.
 
 #### Step 4: Add Test Coverage
-```sql
--- Test parameter
-FUNCTION_NAME(PARAM = @parameter)
-
--- Test outer reference
-FUNCTION_NAME(PARAM = outerref.column)
-
--- Test computed expression
-FUNCTION_NAME(PARAM = value + 1)
+Add tests within the existing test framework:
+```csharp
+[TestMethod]
+public void VerifyGrammarExtension()
+{
+    var parser = new TSql170Parser(true);
+
+    // Test parameter
+    var sql1 = "SELECT FUNCTION_NAME(PARAM = @parameter)";
+    var result1 = parser.Parse(new StringReader(sql1), out var errors1);
+    Assert.AreEqual(0, errors1.Count, "Parameter syntax should work");
+
+    // Test outer reference
+    var sql2 = "SELECT FUNCTION_NAME(PARAM = outerref.column)";
+    var result2 = parser.Parse(new StringReader(sql2), out var errors2);
+    Assert.AreEqual(0, errors2.Count, "Outer reference syntax should work");
+
+    // Test computed expression
+    var sql3 = "SELECT FUNCTION_NAME(PARAM = value + 1)";
+    var result3 = parser.Parse(new StringReader(sql3), out var errors3);
+    Assert.AreEqual(0, errors3.Count, "Computed expression syntax should work");
+}
 ```
 
+**⚠️ CRITICAL**: Add this test method to an existing test class (e.g., `Only170SyntaxTests.cs`). **Never create standalone test projects.**
+
 ### Real-World Example: VECTOR_SEARCH TOP_N
 
 **Problem**: `VECTOR_SEARCH` TOP_N parameter only accepted integer literals.

.github/instructions/parser.guidelines.instructions.md

@@ -59,11 +59,21 @@ case TSql80ParserInternal.Identifier:
 ## Step-by-Step Fix Process
 
 ### 1. Reproduce the Issue
-Create a test case to confirm the bug:
-```sql
-SELECT 1 WHERE (REGEXP_LIKE('a', 'pattern'));  -- Should fail without fix
+Create a test case within the existing test framework to confirm the bug:
+```csharp
+[TestMethod]
+public void ReproduceParenthesesIssue()
+{
+    var parser = new TSql170Parser(true);
+    var sql = "SELECT 1 WHERE (REGEXP_LIKE('a', 'pattern'));";
+    var result = parser.Parse(new StringReader(sql), out var errors);
+    // Should fail before fix, pass after fix
+    Assert.AreEqual(0, errors.Count, "Should parse without errors after fix");
+}
 ```
 
+**⚠️ IMPORTANT**: Add this test to an existing test class like `Only170SyntaxTests.cs`, **do not** create a new test project.
+
 ### 2. Identify the Predicate Constant
 Find the predicate identifier in `CodeGenerationSupporter`:
 ```csharp

.github/instructions/testing.guidelines.instructions.md

@@ -13,6 +13,43 @@ The SqlScriptDOM testing framework validates parser functionality through:
 4. **Version-Specific Testing** - Tests syntax across multiple SQL Server versions (SQL 2000-2025)
 5. **Exact T-SQL Verification** - When testing specific T-SQL syntax from prompts or user requests, the **exact T-SQL statement must be included and verified** in the test to ensure the specific syntax works as expected
 
+## Quick Verification Tests
+
+For rapid verification and debugging, add simple test methods directly to existing test classes:
+
+```csharp
+[TestMethod]
+[Priority(0)]
+[SqlStudioTestCategory(Category.UnitTest)]
+public void VerifyMyNewSyntax()
+{
+    var parser = new TSql170Parser(true);
+
+    // Test basic syntax
+    var query1 = "SELECT YOUR_NEW_FUNCTION('param1', 'param2');";
+    var result1 = parser.Parse(new StringReader(query1), out var errors1);
+    Assert.AreEqual(0, errors1.Count, "Basic syntax should parse");
+
+    // Test complex variations
+    var query2 = "SELECT YOUR_NEW_FUNCTION(@variable);";
+    var result2 = parser.Parse(new StringReader(query2), out var errors2);
+    Assert.AreEqual(0, errors2.Count, "Variable syntax should parse");
+
+    Console.WriteLine("✅ All tests passed!");
+}
+```
+
+**Where to Add Quick Tests:**
+- **SQL Server 2025 (170) features**: Add to `Test/SqlDom/Only170SyntaxTests.cs`
+- **SQL Server 2022 (160) features**: Add to `Test/SqlDom/Only160SyntaxTests.cs`
+- **Earlier versions**: Add to corresponding `Only<version>SyntaxTests.cs`
+
+**When to Use:**
+- Quick verification during development
+- Debugging parser issues
+- Initial syntax validation before full test suite
+- Rapid prototyping of test cases
+
 ## Test Framework Architecture
 
 ### Core Components
@@ -29,6 +66,30 @@ The SqlScriptDOM testing framework validates parser functionality through:
 3. **Validate Phase**: Generated output is compared against baseline file
 4. **Error Validation**: Parse error count is compared against expected error count for each SQL version
 
+## ❌ Anti-Patterns: What NOT to Do
+
+### Do NOT Create New Test Projects
+
+- ❌ **Don't create new `.csproj` files for testing**
+- ❌ **Don't create console applications** like `TestVectorParser.csproj` or `debug_complex.csproj`
+- ❌ **Don't create standalone test runners**
+- ❌ **Don't add new projects to the solution for testing**
+
+### Why This Causes Problems
+
+1. **Build Issues**: New projects often fail to build due to missing dependencies
+2. **Integration Problems**: Standalone projects don't integrate with existing test infrastructure
+3. **Maintenance Overhead**: Additional projects require separate maintenance and documentation
+4. **CI/CD Conflicts**: Build pipelines aren't configured for ad-hoc test projects
+5. **Resource Waste**: Creates duplicate testing infrastructure instead of using established patterns
+
+### The Correct Approach
+
+✅ **Always add test methods to existing test classes**:
+- Add to `Test/SqlDom/Only170SyntaxTests.cs` for SQL Server 2025 features
+- Add to `Test/SqlDom/Only160SyntaxTests.cs` for SQL Server 2022 features
+- Use the established test framework patterns documented in this guide
+
 ## Adding New Tests
 
 ### 1. Create Test Script
@@ -639,6 +700,52 @@ new ParserTest170("ErrorConditionTests170.sql",
     nErrors160: 3, nErrors170: 3),
 ```
 
+## Real-World Example: VECTOR Parsing Verification
+
+This example shows the correct approach used to verify VECTOR data type parsing functionality:
+
+```csharp
+[TestMethod]
+[Priority(0)]
+[SqlStudioTestCategory(Category.UnitTest)]
+public void VerifyComplexQueryFix()
+{
+    // Test VECTOR parsing in various contexts - this is the real bug we found and fixed
+    var parser = new TSql170Parser(true);
+
+    // Test 1: Basic VECTOR with base type
+    var query1 = "SELECT CAST('[1,2,3]' AS VECTOR(3, Float32));";
+    var result1 = parser.Parse(new StringReader(query1), out var errors1);
+    Assert.AreEqual(0, errors1.Count, "Basic VECTOR with base type should parse");
+
+    // Test 2: VECTOR in complex CAST (from original failing query)
+    var query2 = "SELECT CAST('[-6.464173E+08,1.040823E+07,1.699169E+08]' AS VECTOR(3, Float32));";
+    var result2 = parser.Parse(new StringReader(query2), out var errors2);
+    Assert.AreEqual(0, errors2.Count, "VECTOR with scientific notation should parse");
+
+    // Test 3: VECTOR in CONVERT (from original failing query)  
+    var query3 = "SELECT CONVERT(VECTOR(77), '[-7.230808E+08,4.075427E+08]');";
+    var result3 = parser.Parse(new StringReader(query3), out var errors3);
+    Assert.AreEqual(0, errors3.Count, "VECTOR in CONVERT should parse");
+
+    // Test 4: VECTOR in JOIN context (simplified version of original complex query)
+    var query4 = @"SELECT t1.id 
+                   FROM table1 t1 
+                   INNER JOIN table2 t2 ON t1.vector_col = CAST('[1,2,3]' AS VECTOR(3, Float32));";
+    var result4 = parser.Parse(new StringReader(query4), out var errors4);
+    Assert.AreEqual(0, errors4.Count, "VECTOR in JOIN condition should parse");
+
+    Console.WriteLine("✅ All VECTOR parsing tests passed - the original VECTOR bug is fixed!");
+}
+```
+
+**Key Points from This Example:**
+1. Test was added directly to `Only170SyntaxTests.cs` - no new project created
+2. Tests multiple contexts where the syntax appears (CAST, CONVERT, JOIN)
+3. Uses inline assertions with descriptive messages
+4. References the original bug being fixed
+5. Provides immediate feedback via Console.WriteLine
+
 ## Advanced Testing Patterns
 
 ### Multi-File Tests

SqlScriptDom/Parser/TSql/Ast.xml

@@ -4769,6 +4769,38 @@
     <Member Name="ModelName" Type="SchemaObjectName" Summary="Model name used in USE MODEL clause." />
     <Member Name="OptionalParameters" Type="ScalarExpression" Summary="Optional PARAMETERS clause, must evaluate to JSON." />
   </Class>
+  <Class Name="AIAnalyzeSentimentFunctionCall" Base="PrimaryExpression" Summary="Represents the ai_analyze_sentiment built-in.">
+    <InheritedClass Name="PrimaryExpression" />
+    <Member Name="Input" Type="ScalarExpression" Summary="Input text to analyze sentiment for." />
+  </Class>
+  <Class Name="AIFixGrammarFunctionCall" Base="PrimaryExpression" Summary="Represents the ai_fix_grammar built-in.">
+    <InheritedClass Name="PrimaryExpression" />
+    <Member Name="Input" Type="ScalarExpression" Summary="Input text to fix grammar for." />
+  </Class>
+  <Class Name="AIClassifyFunctionCall" Base="PrimaryExpression" Summary="Represents AI_CLASSIFY(...) function call.">
+    <InheritedClass Name="PrimaryExpression" />
+    <Member Name="Input" Type="ScalarExpression" Summary="Variable or column identifier providing the text to classify." />
+    <Member Name="Labels" Type="ScalarExpression" Collection="true" Summary="One or more label expressions (at least two)." />
+  </Class>
+  <Class Name="AIExtractFunctionCall" Base="PrimaryExpression" Summary="Represents the ai_extract built-in.">
+    <InheritedClass Name="PrimaryExpression" />
+    <Member Name="Input" Type="ScalarExpression" Summary="Input text to extract from." />
+    <Member Name="Labels" Type="ScalarExpression" Collection="true" Summary="One or more label expressions." />
+  </Class>
+  <Class Name="AIGenerateResponseFunctionCall" Base="PrimaryExpression" Summary="Represents AI_GENERATE_RESPONSE(...) function call.">
+    <InheritedClass Name="PrimaryExpression" />
+    <Member Name="PromptPart1" Type="ScalarExpression" Summary="First prompt part (required)." />
+    <Member Name="PromptPart2" Type="ScalarExpression" Summary="Second prompt part (optional)." />
+  </Class>
+  <Class Name="AISummarizeFunctionCall" Base="PrimaryExpression" Summary="Represents AI_SUMMARIZE(...) function call.">
+    <InheritedClass Name="PrimaryExpression" />
+    <Member Name="Input" Type="ScalarExpression" Summary="Variable or column identifier providing the text to summarize." />
+  </Class>
+  <Class Name="AITranslateFunctionCall" Base="PrimaryExpression" Summary="Represents AI_TRANSLATE(...) function call.">
+    <InheritedClass Name="PrimaryExpression" />
+    <Member Name="Input" Type="ScalarExpression" Summary="Text to translate; variable or column reference." />
+    <Member Name="Language" Type="ScalarExpression" Summary="Target language; typically string literal or identifier." />
+  </Class>
   <Class Name="VectorSearchTableReference" Base="TableReferenceWithAlias" Summary="Represents the VECTOR_SEARCH table-valued function call.">
     <InheritedClass Name="TableReferenceWithAlias" />
     <Member Name="Table" Type="TableReferenceWithAlias" Summary="Table on which perform the search." />

SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs

@@ -101,6 +101,13 @@ internal static class CodeGenerationSupporter
         internal const string Aggregate = "AGGREGATE";
         internal const string AiGenerateChunks = "AI_GENERATE_CHUNKS";
         internal const string AIGenerateEmbeddings = "AI_GENERATE_EMBEDDINGS";
+        internal const string AIAnalyzeSentiment = "AI_ANALYZE_SENTIMENT";
+        internal const string AIClassify = "AI_CLASSIFY";
+        internal const string AIExtract = "AI_EXTRACT";
+        internal const string AIFixGrammar = "AI_FIX_GRAMMAR";
+        internal const string AIGenerateResponse = "AI_GENERATE_RESPONSE";
+        internal const string AISummarize = "AI_SUMMARIZE";
+        internal const string AITranslate = "AI_TRANSLATE";
         internal const string Algorithm = "ALGORITHM";
         internal const string AlterColumn = "ALTERCOLUMN";
         internal const string All = "ALL";

SqlScriptDom/Parser/TSql/TSql170.g

@@ -19142,9 +19142,21 @@ joinElement[SubDmlFlags subDmlFlags, ref TableReference vResult]
     ;
 
 selectTableReferenceElement [SubDmlFlags subDmlFlags] returns [TableReference vResult = null]
+{
+    IToken tAfterJoinParenthesis = null;
+}
     :
+        // Apply SaveGuessing optimization ONLY when VECTOR keyword is detected in lookahead
+        // This fixes VECTOR parsing in deeply nested JOINs without breaking other valid SQL patterns
+        {ContainsVectorInLookahead()}?
+        ({ if (!SkipGuessing(tAfterJoinParenthesis)) }:
+            (joinParenthesis[subDmlFlags])=> ({ SaveGuessing(out tAfterJoinParenthesis); }:))=>
+        ({ if (!SkipGuessing(tAfterJoinParenthesis)) }:
+            vResult=joinParenthesis[subDmlFlags])
+    |
+        // Standard syntactic predicate for all other cases
         (joinParenthesis[subDmlFlags])=>
-        vResult=joinParenthesis[subDmlFlags]
+            vResult=joinParenthesis[subDmlFlags]
     |   vResult=selectTableReferenceElementWithoutJoinParenthesis[subDmlFlags]
     ;
 

SqlScriptDom/Parser/TSql/TSql170ParserBaseInternal.cs

@@ -70,5 +70,44 @@ protected SecurityObjectKind ParseSecurityObjectKindTSql170(Identifier identifie
                     return TSql160ParserBaseInternal.ParseSecurityObjectKind(identifier1, identifier2);
             }
         }
+
+        /// <summary>
+        /// Checks if VECTOR keyword appears in the upcoming tokens within a reasonable lookahead window.
+        /// Used to determine if SaveGuessing optimization is needed for VECTOR data type parsing.
+        /// </summary>
+        /// <returns>true if VECTOR keyword found in lookahead; false otherwise</returns>
+        protected bool ContainsVectorInLookahead()
+        {
+            // Scan ahead looking for VECTOR keyword (case-insensitive identifier match)
+
+            const int LookaheadLimit = 100; // Define a named constant for the lookahead limit
+            // We scan up to LookaheadLimit tokens to handle deeply nested JOIN structures with VECTOR types
+            for (int i = 1; i <= LookaheadLimit; i++)
+            {
+                IToken token;
+                try
+                {
+                    token = LT(i);
+                }
+                catch (Exception ex)
+                {
+                    Debug.WriteLine($"Error accessing token at lookahead index {i}: {ex.Message}");
+                    break;
+                }
+                if (token == null || token.Type == Token.EOF_TYPE)
+                {
+                    break;
+                }
+
+                // Check if this is an identifier token with text "VECTOR"
+                if (token.Type == TSql170ParserInternal.Identifier &&
+                    string.Equals(token.getText(), CodeGenerationSupporter.Vector, StringComparison.OrdinalIgnoreCase))
+                {
+                    return true;
+                }
+            }
+
+            return false;
+        }
     }
 }