The Data Director for Pimcore Version 3.7 Is Here

Our powerful bundle for exchanging data between Pimcore and external data sources of all kinds has been updated again. Read here what has changed in Data Director version 3.7.

Revised Summary Window

The summary window has been revised to make it easier for users to track the success of imports and exports. It now contains an accordion with three sections:

• Errors
  Including direct links to the affected items.
• Changed elements
  Filterable by elements changed or not changed by the import plus search function when looking for specific elements.
• Result document
  Provides file or result messages as a download.

In this way, data maintainers without technical knowledge can also carry out imports and exports - for example via main menu links or object wizard forms. Once the operation is complete, they receive a clear and visually appealing summary of the results.

Reintroduction of the Admin

In Pimcore 11, the database administration tool has been removed. This was a big disadvantage for Data Director users, as it was no longer possible to view the database to test SQL conditions for dataports. The Data Director now reintroduces this possibility for Pimcore 11.

In addition, some useful functions are added that can also be used in older Pimcore versions (Pimcore 10 and below):

• Fixing the error "Cannot modify header information"
• Auto-suggest for SQL
• Sticky table headers to see the column names when scrolling down
• Added tooltip for displaying the human readable date format for UNIX timestamps

Pimcore Core Improvements

The Data Director bundle not only provides a fast and flexible way to implement imports and exports, but can be used for all Product Information Management (PIM) tasks. That's why Data Director adds some useful (yet overridable) customizations to make day-to-day PIM tasks more efficient. These include:

• Support for the "Back/Forward" browser button to more easily switch between tabs.
• Extension of the context menu for tabs so that tabs can be closed to the left/right of the current tab.
• Added a button to the class definition to start a dataport directly from the editing context.
• Faster deletion of elements: Instead of 2*n consecutive AJAX requests, all selected elements are deleted in a background process (incl. moving the deleted elements to the trash).
• Auto-sync of changed fields if the opened element has been changed by other users/import (if the field value has not been edited by current users).
• Display of recently opened elements as a submenu of the "Recently opened elements" menu icon.
• Translatable keys/names in the element tree.
• Live updates for calculated value fields.
• Changed behavior of the quick search:
  • Only find items that contain ALL search terms
  • Exclusion of files in import archive folders

In the "Pimcore on Steroids" series, we demonstrate the simple activation of these fundamental changes in short demonstration videos on YouTube and go into their concrete benefits.

New Import/Export Functions

• Implementation of a template for filling out a predefined Excel file
• Support for placeholders (*) for FTP/SFTP/FTPS import resources, e.g. ftp://user:password@example.org/import/*.xml
• Support for
  sftp://username:password@example.org/folder/BMEcat_*.xml | latest,
  to use placeholders in FTP URLs, but still retrieve only the latest (= last modified) file
• Addition of the Data Query Selector "latestVersionWith".
  In review and approval workflows, objects can have several states at the same time: Initially, the object's data is not approved. This can be modeled with a checkbox or a selection field. After the check, the status is set to "approved". Later, the data is updated and the status is set to "not approved" again. At this point, however, it may still be necessary to access the last approved data. This can be done with the data query selector "latestVersionWith". For example, to retrieve the last approved name of the object, you can use latestVersionWith#status,approved:name:
  • latestVersionWith#status,approved retrieves the latest version that has the status =approved The name is retrieved from this version.
  • In this way, you can create exports that only export approved data.
  • If you want to use a different comparison operator, use the third parameter, e.g. latestVersionWith#validUntil,2024-01-01,>=.
• Support for strings as import resources such as { "sku": , image: } for a JSON import and dynamic filling of placeholders via parameters.

UI Improvements

• New status panel to get an overview of all data port runs (similar to the history panel, but for all data ports).
• History panel:
  • Faster loading.
  • Data port logs with > 100 MB are made available as a download instead of being displayed in the browser.
  • In the event of an error, only the error message is displayed in the tooltip of the log link, not the entire code of the callback function.
  • Display of parameters as import resource for queue runs of parameterized dataports.
  • Moved the "Extract raw data" and "Process raw data" buttons to the drop-down menu of the "Start complete import/export" button.
  • Dynamically show and hide the "Queue Processor Monitor" button.
• Preview panel: Scrolling in the log window is improved when processing individual raw data
• Dataport search: Support for keyword-based search so that the search for "Import products" also finds the dataport "Import products from ERP".

Performance Improvements

• If the callback function returns object/array, the entire object is not serialized, but Importer::getLogOutput() This means that objects with many related elements do not have to be loaded separately for the preview.
• Values for fields of the type "Calculated value (Data Query Selector)" are stored temporarily.
• Automatic exports are no longer queued if the raw data has not been changed since the last run. This means that there are fewer elements in the queue.

Minor Changes

• Added the Data Query Selector fields to retrieve field definitions of classes and object blocks including all their settings.
• Recognition of changes to field collections: If the same field collection elements are present after the import, but only in a different order, they are not saved.
• Support for the translation of objectbrick/classification memory field titles via objectbrick:labels#en.
• OpenAI integration: Support for deriving values for the field type "country multiselect".
• DeepL translation: Support for the automatic use of glossaries.
• Display of the current DD version in the Pimcore icon tooltip at the bottom left. Otherwise, there is no way to see the installed version as the bundlespanel has been removed from the backend.
• Added callback function template to convert HTML to Markdown.
• Transpile SQL condition: For the source data class "Asset", direct access to properties via propertyName='value' is now supported.
• Bugfix: If an object was saved twice in the same second, the queue processor sometimes did not start the automatic dataport because the first version did not match the SQL condition of the dataport and the second version was not checked.
• Support for importing localized block fields via #locale syntax, e.g. ['fieldName#en' => 'value'].
• Support of the field type "Extended many-to-many relationship" as a key field.
• Streamed XML parsing: better tolerance if the same tag appears in the data at different levels, e.g. <product><related_products><product></product></related_products></product>.
• Mapping panel: Separation of "other mapped fields" and "environment variables" in "available variables".
• Attribute mapping: If the field type is not supported as a key field, 1=0 is now used as SQL condition and a warning is logged. Previously, this field was skipped without adding a condition, so many records not matching the condition were found.
• Support of PHP code for calculated value fields.
• Support for importing the field type "Table".
• Support for reading UTF-16 XML files.
• Support for starting the object wizard via a button and pre-filling the matching fields with data from the current object.
• Support for AWS S3 as an import resource.
• Moving the Dataport version files to the Pimcore version directory for easier adding of var/bundles/BlackbitDataDirectorBundle to Git.
• Performance improvement for zip exports.
• Support for importing data into dachcom-digital/seo bundle.
• When executing automatic dataports, do not check the "View" authorization, otherwise an inconsistent state could occur:
  • User only has access to class A
  • User changes object of class A
  • automatic dataport for source class B is started for objects that depend on the currently saved product of the class

Previously there was an error here because the triggering user did not have authorization for class B. Now the Dataport is executed in the event of an inconsistent status.

Update to Version 3.7 Now!

You can easily download the new version directly via your personal BitBucket account in the Blackbit update subscription. If you have any questions, please contact us directly.

Helpful Video Tutorials for the Pimcore Data Director

We offer detailed instructions and many practical tips on the efficient use of the Pimcore Data Director in the video tutorials in the Blackbit Academy and on the Blackbit YouTube channel - there you will also find all episodes of our new series "Pimcore on Steroids" with productivity-enhancing tips for your daily operations in the Pimcore backend.

Not Yet Familiar With the Data Director Bundle?

If you would like to get to know our powerful import and export bundle better first, why not try it out in our free demo installation?

Would you like to test a specific use case? Then please send us your task and data. We would be happy to offer you a workshop in which we show you how to solve your individual requirements with the Data Director.