3.5.0 - Pimcore Data Director in the new version

Our developers have added new practical functions to the new version of the Pimcore Data Director. The biggest change: The Data Director is now fully compatible with Pimcore 11.

Pimcore 11 compatibility

Version 3.5.0 is fully compatible with Pimcore 11: Pimcore has removed some bundles from the core product (Application Logger, Reports, Redirects etc.). However, the Data Director is no longer dependent on these bundles and thus follows the Pimcore standard: Simply install the bundles you need - the Data Director works with all your installed bundles.

We have also improved compatibility in the other direction: Data Director 3.5 is compatible with PHP 7.3, so Pimcore 5 and 6 are still supported.

Background process to check whether automatic dataports should be executed

In previous versions, a check was made directly when saving a Pimcore element to determine which automatic dataports should be started. If an automatic dataport had a complex SQL query, this could take some time. This also applied to manually executed saves in the Pimcore backend, so in this case the user had to wait until the check for all automatic dataports was complete. In version 3.5.0, this check is now performed in a background process so that manually saving an element is just as fast as without Data Director.

Automatic testing of data ports

With the command bin/console data-director:create-test <Dataport ID> you can create a PHPUnit test for the specified dataport. A test class is created based on the currently available raw data and the mapping. You can execute these automatic tests via PHPUnit.

Main purpose of the automatic tests:

1. Ensure that everything works as before after changes to the dataport settings or attribute mapping.
2. Ensure that before and after updates/changes to the underlying libraries such as Pimcore or Data Director itself, everything works as before.

A phpunit.xml is required to run the tests. If this does not already exist in the Pimcore root directory, it will be created automatically. If it already exists, a new test suite will be added. Once the tests have been created, you will immediately receive help on how to execute the tests.

Dataport synchronisation between multiple Pimcore systems

The command data-director:deployment:dataport-rebuild can now also be used to synchronise dataports between multiple Pimcore systems. There are two new REST endpoints:

• GET http(s)://<YOUR-DOMAIN>/api/dataports?apikey=<API key>
  returns a JSON list of dataports that the user specified by <API Key> is allowed to configure.
  
  
• GET http(s)://<YOUR-DOMAIN>/api/dataports/<dataport id or name>?apikey=<API key>
  returns the dataport configuration as JSON, which can be used to set up dataports via the data-director:deployment:dataport-rebuild command (just like when exporting the dataport configuration or using Git to save JSON configuration files).

To import dataports from a remote system, you can call
bin/console data-director:deployment:dataport-rebuild --source=https://other-pimcore.com --api-key=<API key>
. As an optional argument, you can specify the name of the dataport to be retrieved - otherwise all dataports will be retrieved.

If you are working with a remote source, only dataports whose last version was imported or which do not yet exist will be imported. For existing dataports whose last version was edited by a real user, you will be asked whether you want to overwrite the local dataport configuration to prevent local changes from being overwritten. As soon as you allow the import, the last version will be marked as imported and can therefore be synchronised automatically in the future. If you always want to overwrite local changes, use the --force flag.

UI changes

Pimcore perspectives

Perspectives are a powerful but rarely used feature of Pimcore. The Data Director now offers a basic perspective based on the groups of data object classes. All data object classes that belong to the same group are displayed in the same perspective panel. The root folder is automatically optimised to the lowest parent object that contains all objects of the corresponding class.

In addition, the perspective also contains a dashboard that includes a queue monitor and portlets for tagged elements and data port errors. Of course, these portlets can also be used in other dashboards (e.g. the portlet for tagged elements could be used as a to-do list).

Element Tree Tooltips

By default, Pimcore only displays the element ID and class when hovering over an element in the tree. This information is rarely useful. The Data Director in version 3.5.0 now comes with a standard tooltip that displays all fields that are configured to be displayed in the grid view in the tooltip.

Path formatter for assets

@DataDirectorSearchViewPathFormatter and @DataDirectorGridViewPathFormatter now support the display of asset metadata for asset-based relations (as a tooltip when hovering over them with the mouse pointer).

They also now support direct editing with MS Office tools for Excel, Word and PowerPoint assets: The displayed asset path is linked: If you click on it, the asset will be opened in MS Office. All saved changes are transferred directly to Pimcore via the WebDAV interface - without having to manually download and upload the file again.

Dataport settings

• Support for copying multiple raw data fields
• Simplified access to complex import resources:
  • Support for direct input of PHP/shellcode, e.g. for accessing data from OAuth2 APIs. This eliminates the need to create a separate script file.
  • Support for using PHP code as an import resource that returns a URL (e.g. HTTP, FTP, Pimcore asset path). This allows the script to focus on the logic to generate the URL, while the code to actually access the resource is delegated to the Data Director.
• Support for querying asset metadata fields in SQL conditions using their names, e.g. metaField=1.
• Support for placeholders in the target folder/asset target folder (e.g. when specifying the target folder in dependent data port parameters).

Attribute mapping

• Improved performance of the attribute mapping panel.
• Callback function window
  In the tree of available variables, individual raw item fields are displayed as a subtree of $params['rawItemData'] The same applies to individual fields of $params['currentObjectData'].
• The data port version is now saved first, then attribute mapping sample data is processed. This means that you can still revert to the previous version in the event of critical syntax errors in one of the callback functions.
• Fields that have been mapped using the "automatic mapping" button are no longer saved automatically, but the mapping result can be checked and then saved.
• Addition of a checkbox for the "Path" field to delete subordinate elements before import, similar to "Clear before import" for relational fields.

Other changes

The following additional functions are included in 3.5:

• SQL condition
  Support of filtering for metadata columns of extended many-to-many object relationships, e.g. active=1 for an extended many-to-many relationship "markets" with boolean meta field "active".
• Fewer file copies are now created for remote data port resources.
• Added result document template for Excel export.
• Identical raw data elements are no longer processed multiple times when running dd:process.
• Support for importing quantity values with thousands separators such as 1,234.56 kg.
• Enhanced logging: Adding logs to the dataport log, which is logged via \Pimcore\Logger::error() and similar methods.
• Support for inferring field values from text via the ChatGPT API for advanced many-to-many relationships.
• Changing the priority for the resolution of virtual fields, e.g. :
  • if the field "field" really exists in the target class, the return value of this field is used (to reuse return values of other fields)
  • Use query parameter "field
  • Use raw data field with the name "field"
  • Previously, there were problems with dependent dataport parameters if the called dataport had a raw data field with the same name. Then there was no way to access the parameter.
• Serialisation bugfix
  Export of boolean values as 0/1 was "true"/"false" in some cases.
• Only provide environment variables that are actually used in callback functions to reduce memory usage.
• Support of SQL query optimisations for localised relation fields such as relationField#de LIKE '%,1234,%'. Previously, only non-localised fields were supported here.
• Support for accessing the localised classification store via the data query selector classificationStore#en.
• Data query selector for classification stores: Syntax classificationStoreField:groupName now returns a key value array of all fields of the group and classificationStoreField:groupName:fieldName returns the value of the specified classification store field.
• Support for IN() in data query selectors, e.g. Product:sku:[123,234]
• Automatically run Pimcore-based dataports when using the initialisation function "Run every x hours" (triggers a pseudo cronjob).
• Export thumbnails in WEBP format if the thumbnail format is "Auto (Web optimised)"
• Support for the automatic creation of non-existent selection field options for classification memories and for multi-select fields.
• Raw data extraction
  No empty raw data fields are imported, which allows faster raw data extraction, e.g. CSV with technical data where certain columns only apply to products of a certain product group but are empty for all others.
• Automatic deletion of (temporarily) created superordinate elements that no longer have any subordinate elements. This happens when the import sequence does not create parent items first.
• Serialisation of the source/target configuration of the dataport as JSON instead of serialize() for better readability or diff view for version files.
• Queue monitor
  Saves current status in the database instead of file-based. This is necessary if the queue processor runs in a container other than Pimcore.
• Many other bugfixes.

Useful Pimcore Data Director tutorials

We offer many more helpful tips and detailed instructions on how to use the Pimcore Data Director in the video tutorials in the Blackbit Academy and on the Blackbit YouTube channel.

Not yet familiar with our Data Director?

If you would like to get to know our powerful import and export bundle first, then visit our shop now to find out what possibilities the Data Director for Pimcore opens up for you.
Contact us now and we will be happy to show you the bundle in a personalised demo.