BulkCopy in mssql-python - API Spec & Discussion

[bewithgaurav]

Discussion for Bulk Copy in mssql-python (Issue #18)

We’re adding a BulkCopy API (cursor.bulkcopy()) to mssql-python for fast, high-throughput inserts into SQL Server.

Due to all the interest, this discussion thread is here for us to understand from you, about your usage scenarios - feel free to comment your suggestions and propose improvements!

If you’re using (or planning to use) bulk copy, feel free to comment with:

• Your use case (ETL, migration, backfill, analytics, etc.)
• Options you ended up using (batch_size, table_lock etc.)
• Anything that felt unclear or surprising

BulkCopy API Spec

The cursor.bulkcopy() method provides bulk data loading to SQL Server, similar to ADO.NET’s SqlBulkCopy.
It follows the standard DB-API cursor pattern and uses keyword arguments for configuration.

---

Method Signature

cursor.bulkcopy(
    table_name: str,
    data_source: Iterator[Tuple],
    **kwargs
) -> Dict[str, Any]

Full Signature with All Available Keyword Arguments

cursor.bulkcopy(
    table_name: str,
    data_source: Iterator[Tuple],

    # Performance settings
    batch_size: int = 1000,
    timeout_in_seconds: int = 30,

    # Column mapping
    column_mappings: Optional[List[Tuple[Union[str, int], str]]] = None,

    # Bulk copy options
    keep_identity: bool = False,
    check_constraints: bool = False,
    table_lock: bool = False,
    keep_nulls: bool = False,
    fire_triggers: bool = False,
    use_internal_transaction: bool = False
) -> Dict[str, Any]

---

Parameters

Required Positional Parameters

• table_name (str)
  Name of the destination table.

• data_source (Iterator[Tuple])
  Iterator yielding tuples of row data.

---

Keyword Arguments

All parameters beyond table_name and data_source must be passed as keyword arguments.

---

Performance Settings

• batch_size (int, default: 1000)
  Number of rows per batch.

  • Smaller values: lower memory usage, more network round trips
  • Larger values: higher memory usage, better throughput
  • Recommended range: 1000–10000

• timeout_in_seconds (int, default: 30)
  Operation timeout in seconds.

  • Set to 0 for no timeout
  • Increase for very large datasets

---

Column Mapping

• column_mappings (List[Tuple], optional)
  Maps source columns to destination columns.

  Format:

  [(source, destination), ...]

  • source: column name (str) or column index (int)
  • destination: destination column name (str)

  Example:

  [('src_id', 'UserID'), (0, 'FirstColumn')]

---

Bulk Copy Options

• keep_identity (bool, default: False)
  Preserve source identity values.
  Equivalent to SET IDENTITY_INSERT ON.

• check_constraints (bool, default: False)
  Enforces table constraints during bulk copy.
  Disabled by default for performance.

• table_lock (bool, default: False)
  Acquires a table-level lock to reduce locking overhead.
  Recommended when exclusive access is available.

• keep_nulls (bool, default: False)
  Preserves explicit NULL values instead of applying column defaults.

• fire_triggers (bool, default: False)
  Fires INSERT triggers during bulk copy.
  Disabled by default for performance.

• use_internal_transaction (bool, default: False)
  Wraps the bulk copy operation in an internal transaction.
  Automatically rolls back on error.

---

Return Value

Returns a dictionary containing operation statistics:

{
    'rows_copied': 10000,       # Number of rows successfully copied
    'batch_count': 10,          # Number of batches processed
    'elapsed_time': 1.5,        # Total elapsed time (seconds)
    'rows_per_second': 6666     # Throughput
}

Looking forward to your feedback!

cc: @dlevy-msft-sql @saurabh500 @sumitmsft

[saurabh500]

Re

Column Mapping
...
column_mappings (List[Tuple], optional)
source: column name (str) or column index (int)

I think we dont have a usecase for column name (str) in the source.
We may end up limiting the source to only support column index(int)

[amachanic]

Agree, column name in the source is not useful, unless you decide to support mapping collections there - which I also don't think is a necessary thing.

What I would ideally like to see here personally is a slightly simpler implementation: Just let the user pass a list of destination column names, in order, that correspond to the input tuples. So if my input tuple has three elements corresponding to colB, colA, and colC respectively, I'd like to pass in ['colB', 'colA', 'colC'], rather than [(0, 'colB'), (1, 'colA'), (2, 'colC')].

[amachanic]

I'd like to understand the reasoning behind hiding so much behind the kwargs instead of using optional parameters. I feel that this really makes the interface much more difficult to use. (The various options aside from the column mappings -- which I feel should be a first-class thing -- could all be grouped in a named tuple or class with defaults, if you want to cut it down to just two optional params.)