We Make the Comparison: Pimcore Data Director vs. Data Hub

With the Data Importer as an extension for the Data Hub Bundle, Pimcore offers its own tool for importing and exporting data and connecting external data sources. But can Pimcore's own solution keep up with Blackbit's Data Director? We make the comparison.

In our comparison, we look at the pure performance of the two import and export bundles on the one hand, and their range of functions and usability on the other - two criteria that have proven to be critical for us time and again during the development and continuous improvement of the Data Director and in many PIM projects.

Data Director vs. data hub

A direct comparison of the two import and export bundles for Pimcore reveals:

• Better performance when importing the data source
  • Streaming is particularly noticeable when importing XML and JSON. The Data Director streams these files (the entire XML or JSON tree is not read into RAM before the import can begin, but the individual data records are processed one after the other). This significantly reduces RAM consumption, even for XML files several 100 MB in size, such as when importing BMEcat.
• The storage logic of the Pimcore Data Hub is not designed for mass imports

• • For example, to save relations, all elements are first removed from the database and then reinserted. The Data Director, on the other hand, first calculates the difference and then performs the necessary database operations to obtain the desired state.
  • Pimcore's storage logic saves each object individually. With imports, however, it is often the case that similar data operations have to be performed for many objects. The Data Director therefore combines these operations, which means that far fewer SQL operations are required.
  • The adapted storage logic of the Data Director is of course still fully compatible with Pimcore - both in terms of the final state and triggered events.
• Detectability of whether data has changed
  • The Data Director automatically recognizes whether data has changed. If not, the object does not need to be saved again. As a result, imports run faster and the version history of your data objects is not overloaded with a large number of versions where no actual data change has taken place.

• Transparency of imports and exports
  • All import logics are configurable in the Pimcore backend.
  • The data hub often reaches the limits of the possibilities configurable via UI for individual logics and must first be provided with a corresponding extension by a programmer. This results in a non-transparent black box for parts of the import.
    With the Data Director, such logics can be implemented directly in the Pimcore backend. For frequently occurring use cases, transformations are offered that work out-of-the-box, but they can also be adapted or developed completely individually. These individual logics can be implemented as PHP code directly in the Pimcore backend.
  All changes to data ports are versioned, so you can revert to a previous state at any time or compare the current state with the previous version.

• Complete traceability of data changes
  The Data Director offers the following advantages here:

• An import archive documents which import file was responsible for a specific data change.
• Automatic notification of the occurrence of errors, e.g. for cronjob imports that are not constantly monitored by employees.
• Reversibility: Individual data object fields can always be reset to a state in the past. This function proves to be a real savior when, two weeks after an import, it becomes apparent that data in many data objects is now incorrect, but no database backup can be imported at this time, as otherwise the changes made to other fields in the meantime would be lost.

• Importing field collections, blocks and classification memories
  This is generally not possible with Data Hub. The Data Director is not only characterized by this function, but also offers:

• dynamic field mapping for object modules, classification memories and field collections: Specification of values to be imported as a key-value array. If a column is added to the import source, it can be imported automatically without having to adjust the import. This is particularly useful if the leading system for the technical data is not Pimcore and it is therefore still unclear which fields will be delivered from the source system in the future.
  New fields in object modules or classification memories can even be created automatically if desired.
• In addition, selection field options and quantity-value units can be created automatically.
• Event-based triggering of imports and exports
  It is easy to link imports and exports to specific, predefined events - for example:
  • starting an import as soon as a file has been loaded into a specific folder
  • or as soon as an image is uploaded to the Pimcore assets, so that it is automatically assigned to the relevant article based on the file name (transformation from file name to article number can be customized).
  • Or starting an import as soon as an object of a certain class has been saved.
  • "Imports" can also be made on the basis of existing data
    This allows automation to be implemented in Pimcore without much effort. For example for:
    • Automatic translations
      After entering a German text, it is automatically translated into other target languages without having to be started manually.
    • Conversion of workflows
      As soon as an English translation has been maintained, the "checked" checkbox is unchecked and the "last change by" field is automatically set to the current user.
    • Built-in interfaces to DeepL/OpenAI
      Integrated APIs to external service providers in the Data Director enable translations, text generation or feature extraction (reading individual features from a text).
    • Chaining of data ports
      After the end of an import, a further import or export can be triggered automatically.
  • Connection from InDesign to Pimcore
    • The Data Director is particularly useful for connecting Pimcore to InDesign, especially for agencies and marketing departments of larger companies. This allows the entire logic to be implemented in the Pimcore backend, while the graphic design is done in InDesign. Alternative solutions often require a cumbersome workaround in which, for example, HTML has to be adapted in InDesign or entire logics have to be implemented via XSL in order to access the Pimcore database directly.
      This quickly complicates the workflow enormously, as the developers who could program this solution usually do not have InDesign, while the graphic designers are often overwhelmed with such technical details.
      With the Data Director, there is a clean separation of data and design. The exact workflow for connecting Pimcore to EasyCatalog can be found in our detailed YouTube tutorial on connecting EasyCatalog.

  A complete and detailed list of the functions that the Data Director serves better than Pimcore's in-house solution Data Hub for certain purposes can be found in the Pimcore Marketplace under "Advantages compared to other Pimcore import plugins".

  Our conclusion

  Both Data Hub and Data Director fulfill essential tasks in the management of product information. The Data Director offers better performance for many data operations and is already designed to handle various functions, such as the automation of imports and exports or entire workflows. Depending on the focus of the application, both bundles are suitable for your PIM project. Overall, however, the Data Director seems to have been developed closer to real use cases, which quickly becomes noticeable when working with data on a daily basis.

  You don't know our Data Director yet

  If you would like to get to know our powerful import and export bundle, then find out now in our shop what possibilities the Data Director for Pimcore opens up for you.
  Would you like to see the usability and functions of the Data Director for yourself in a live demo? Then contact us now and we will give you a practical demonstration of the bundle in an individual demo.

  Our Pimcore Data Director tutorials

  Are you already working with Data Director? If so, we have lots of helpful tips and detailed instructions on how to use the bundle even more efficiently in the Blackbit Academy video tutorials and in Blackbit's YouTube channel.