Data management entity's exports empty files i.e. if no data to export still it exports a blank file.
Problem statement: Assume 30 interfaces (Per legal entity) scheduled to export every minute, which will result in huge export history per hour. i.e. (30 * No of legal entity * 60 minutes) Manually finding 1 execution where exactly the data is exported in next to impossible, space usage on blob and for integrating application like BizTalk directly consuming these messages using Dequeue involves execution cost.
As mentioned above this has issue at two levels especially for Recurring jobs.
- In the recurring data jobs execution history it is impossible to find the execution with data is next to impossible.
- Integrating application like BizTalk directly consuming these messages using Dequeue involves execution cost.
Potential solution/design: Having a condition to check if records/record count is greater than 0 will restrict the exporting of empty files. This condition should be controlled by the user. Ex: Like skip staging option on data project there should be another parameter to allow user to choose export a file only if it has data. This way it will be more flexible and suitable for any customer scenarios. This resolution at the core level of API’s will control the flow of the messages with minimal change and simple design to address the above problems.