Data Collector version 3.22.x, released in March 2021, includes the following new features and enhancements:

Stage Enhancements
  • JDBC Lookup processor – The processor includes a new Validate Column Mappings property. When selected, all columns listed in the Column Mappings list must exist in the database when the pipeline starts.

  • JDBC Multitable Consumer origin – You can select Square Brackets for the Quote Character property. You can also select None when reading from an Oracle or PostgreSQL database. Previously, this was only possible for MySQL and Microsoft SQL Server databases.

  • Kafka stages – The Use SASL Mechanism as PLAIN checkbox has been renamed SASL Mechanism and provides a menu of options.

  • MongoDB stages – When using LDAP authentication, you can configure the Authentication Mechanism property to select server-driven authentication or plain authentication.

  • MySQL Binary Log origin – The origin includes the following new properties:

    • JDBC Connection property – Enables specifying a full JDBC connection string, along with optional parameters. This property replaces the Hostname and Port properties. Upgraded pipelines are not affected.
    • Use Credentials property – Select to specify a user name and password on the Credentials tab.
  • Oracle CDC Client origin:

    • You can now use the PEG Parser in production. It is no longer considered a beta feature.
    • Starting in version 3.22.2, the origin generates the following CDC record header attributes:
      • oracle.cdc.precisionTimestamp
      • oracle.cdc.redoValue

      • oracle.cdc.sequence.internal
      • oracle.cdc.sequence.oracle
      • oracle.cdc.undoValue

    The PEG Parser can now be used in production. It is no longer considered a beta feature.

  • Salesforce stages:

    • Salesforce stages now support connecting to Salesforce through OAuth. Data Collector uses the Salesforce implementation of JWT Bearer Flow for machine-to-machine OAuth.

    • A new Authentication Method property allows selecting OAuth or basic authentication.

  • Salesforce origin – The Replay Option property is now available when the origin processes change or platform events. Previously, it was only available when processing platform events.

  • SFTP/FTP/FTPS Client stages – You configure the Protocol property to explicitly state the protocol that you want to use. Previously, this information was inferred from the specified resource URL. This does not affect upgraded pipelines.

Connections when Registered with Control Hub

When Data Collector version 3.22.x is registered with Control Hub cloud or with Control Hub on-premises version 3.19.x or later, the following stages support using Control Hub connections:

  • MySQL Binary Log origin

  • PostgreSQL CDC Client origin

  • SFTP/FTP/FTPS Client origin, destination, and executor

Additional Enhancements
  • New delimited data parser – You can configure a CSV Parser property to enable stages to use the Univocity parser to read delimited data. The Univocity parser can enhance pipeline performance when reading wide delimited files. By default, stages use the Apache Commons parser. Upgraded pipelines are not affected by this change.
  • Support bundles – When downloading a support bundle, you can no longer include pipeline snapshots or generated statistics. This is to further protect your data privacy.