docs-spark-connector/source/release-notes.txt at 55ed099ef412e15bcdaaa18c166579edcce8f737 · jordan-smith721/docs-spark-connector · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
=============
Release Notes
=============

MongoDB Connector for Spark 10.3
--------------------------------

The 10.3 connector release includes the following new features:

- Added support for reading multiple collections when using micro-batch or
  continuous streaming modes.

  .. warning:: Breaking Change

     Support for reading multiple collections introduces the following breaking
     changes:

     - If the name of a collection used in your ``collection`` configuration
       option contains a comma, the
       {+connector-short+} treats it as two different collections. To avoid
       this, you must escape the comma by preceding it with a backslash (\\).

     - If the name of a collection used in your ``collection`` configuration
       option is "*", the {+connector-short+} interprets it as a specification
       to scan all collections. To avoid this, you must escape the asterisk by preceding it
       with a backslash (\\).

     - If the name of a collection used in your ``collection`` configuration
       option contains a backslash (\\), the
       {+connector-short+} treats the backslash as an escape character, which
       might change how it interprets the value. To avoid this, you must escape
       the backslash by preceding it with another backslash.

  To learn more about scanning multiple collections, see the :ref:`collection
  configuration property <spark-streaming-input-conf>` description.

MongoDB Connector for Spark 10.2
--------------------------------

The 10.2 connector release includes the following new features:

- Added the ``ignoreNullValues`` write-configuration property, which enables you
  to control whether the connector ignores null values. In previous versions,
  the connector always wrote ``null`` values to MongoDB.
- Added options for the ``convertJson`` write-configuration property.
- Added the ``change.stream.micro.batch.max.partition.count`` read-configuration property,
  which allows you to divide micro-batches into multiple partitions for parallel
  processing.
- Improved change stream schema inference when using the
  ``change.stream.publish.full.document.only`` read-configuration property.
- Added the ``change.stream.startup.mode`` read-configuration property, which specifies
  how the connector processes change events when no offset is available.
- Support for adding a comment to operations.

MongoDB Connector for Spark 10.1.1
----------------------------------

- Corrected a bug in which aggregations including the ``$collStats`` pipeline stage
  did not return a count field for Time Series collections.

MongoDB Connector for Spark 10.1.0
----------------------------------

- Support for Scala 2.13.
- Support for micro-batch mode with Spark Structured Streaming.
- Support for BSON data types.
- Improved partitioner support for empty collections.
- Option to disable automatic upsert on write operations.
- Improved schema inference for empty arrays.
- Support for null values in arrays and lists. The Connector now writes these values
  to MongoDB instead of throwing an exception.

See `this post <https://www.mongodb.com/blog/post/introducing-mongodb-spark-connector-version-10-1>`__
on the MongoDB blog for more information.

MongoDB Connector for Spark 10.0.0
----------------------------------

- Support for Spark Structured Streaming.