Blackbits Data Director for Pimcore: Version 3.1 is here

Busy, busy! After the big jump with version 3.0, our developers have already added some improvements to the in-house import and export bundle. Blackbit presents the latest version of the Data Director - with the following functions and features:

Data Query Selector Grid Operator

With the Pimcore standard grid operators it is sometimes difficult to extract the desired data. It is now much easier with our new Data Query Selector Grid Operator: Extract data from displayed objects, their relationships or even completely different objects. The operator also supports the auto-complete feature, which you may already know from implementing data query selectors in export raw data fields.
In addition, there is now also a new field type "Calculated Value (Data Query Selector)" to define logic at the field definition level (similar to Symfony Expression Language, but with Twig syntax and data query selectors, you have many more options to choose from). This way, you can define the logic of fields with calculated values directly in the field definition.

Performance improvements

Bulk inserts are now used when extracting and processing raw data, instead of storing raw data or data objects individually as before. This results in a performance improvement of about 600% for raw data extraction.

Another performance improvement: virtual fields of the result creation function are now processed only once. Previously, this was done over and over again for each raw data element.

Improved ad-hoc grid exports

Data Director now provides out-of-the-box CSV, XML, and JSON exports of data displayed in the current grid - including custom table configuration and operators - without the need to create a separate data port for this purpose. All processing is server-side, reducing export time compared to Pimcore's standard CSV/Excel export.

Versioning of Dataport settings

All changes to Dataport settings are now versioned. This allows you to track, compare and undo changes for up to 90 days.

Smaller features and improvements

Queue Processing

• Imports with the "automatically run on new data" checkbox where changes to the data source cannot be detected by events (such as file system imports, FTP, URLs, etc.) are now run along with the maintenance cronjob.
• The number of parallel processes allowed in queue processing is saved for each dataport so that the queue processor does not have to restart with very few parallel processes after restart.
• Keeping the scrolling position in the queue monitor
  

Debugging and logging

• Output and logs during the dataport run are displayed not only in the log but also in the preview window for simplified debugging.
• Support for logging complex values with $params['logger'] in callback functions saves the need to convert them to a string first.
• Logging of parsed values from callback functions, e.g. when a callback function returns a data query selector, the object found, if any, is also logged.

Imports and exports

• If the "Enable Inheritance" checkbox is unchecked for Pimcore-based dataports, the SQL condition is now also applied to non-inherited data tables.
• Storing results from DeepL or other translation providers in shared translations to avoid having to retrieve and pay for them again after clearing the Pimcore cache. Previously, these were stored in the Pimcore cache, so the translation had to be paid for again for the same input when the cache was cleared.
• If the data query selector cannot find an object, null is returned. Previously false or even the data query selector string itself was returned.
• Prevent sending an error log email if another error log email has already been sent for the current dataport within 5 minutes.
• Support for field#locale to import data into classification store fields by assigning a callback function for the classification store or object brick container.
• Support for multiple classification memory groups.
• Support for retrieving data from block elements, e.g. Block:each:(name) or Block:each:(relation:name).
• Byte order marker added to CSV export template to correctly display Excel special characters.
• Added new import source type 'fixed length file'.
• Dataport::isQueued() now returns false if queued jobs for this dataport are currently being processed.

User interface

• UX improvement
  Dataport panel: support for automatic creation of raw data fields by uploading a sample import file. The sample file no longer needs to be set as an import resource for this.
  
• Attribute mapping
  If $params['value'] is called in the callback function but no raw data field has been assigned, a warning is now displayed
  .
• History panel
  • Jobs that have been started but are still in the queue are now displayed only 1 × in the history panel. Previously, irritation arose because it suggested that two processes existed, one running and one queued.
  • Support for filtering by "done" and "total" items processed in the dataport run, e.g.
    • done > 0
    • done > 1000
    • total > 1000
    • done < total
    • done = total
  • Logs or history is no longer cleared when raw data or dataport resources are deleted, e.g. to know when a dataport was last executed.
• Preview panel
  • When editing a single raw data item in the preview window, the log window is maximizable.
  • Improved performance in search.

Other improvements

• Symfony Firewall for REST API is replaced with custom token validation. This allows easier installation without having to change global security.yaml.
• Priority is changed to SMALLINT to correctly display the order of raw data fields when there are more than 127 raw data fields in a dataport.
• Added composer script to perform bundle migrations automatically.
• Clean implementation of asynchronous dataport runs. Useful side-effect is to work around issues with non-overridable max_execution_time or proxy timeouts.
• Wait time for import files in a folder or glob expression reduced to 2 seconds (previously 10). This is the amount of time a file must not have been modified to be used in an import. This way, files that are in the process of being uploaded will not be imported.
• The favorite dataport is displayed as a submenu item of the main Data Director menu item if the user cannot see the main Settings menu.

Data Director tutorials on YouTube

For more helpful tips on using the Pimcore Data Director, check out our YouTube video tutorials. Detailed instructions can be found on our YouTube channel.

Not yet familiar with the Data Director?

Curious about our import and export bundle? Then get to know the Data Director: We will show you in an individual demo which possibilities the Data Director offers you. Contact us now!