Integrity Release Notes

From C3 Integrity
Jump to: navigation, search

3.8

  • Performance improvement - Record storage batching
  • Performance improvement - Faster row storage medium
  • Performance improvement - Faster single record edit page generation
  • Bug fix - Storage expiration now works correctly
  • Enhancement - Versions show which user caused each system change
  • Enhancement - Improved browser testing
  • Platform security patches
  • Database adapter security patches
  • Enhancement - Search qualifier form looks the same as any other qualifier form
  • Bug fix - Qualifier multi-select works correctly
  • Enhancement - XML dataset formats support XQuery functions
  • Add support for secure LDAP
  • Update to Rails 3.2.19 (security fixes)
  • Enhancement - API change to track UploadAttemptID and AuditID separately
  • Migrate from upload_attempt_id to audit_id to include undo attempts
  • Note: this section is incomplete

3.7

3.7.6

Bug Fixes

  • Fix expression validations failing when the Dataset's natural key contains date attributes

3.7.5

Bug Fixes

  • Reduction in the number of queries performed when exporting error summaries, this will speed up exporting a large number of errors

3.7.4

Bug Fixes

  • Remove in-memory store step introduced in 3.7.3, this reduces the time needed for a large number of warnings significantly

3.7.3

Bug Fixes

  • When inserting errors into the database, batch all insert statements, this reduces time spent after validation when there are a large number of warnings

3.7.2

Bug Fixes

  • Natural key validation now stores computed information in batches, this reduces time spent during validation with a large natural key set

New Features

  • Record History - When viewing dataset search results, you can access a record's audit history by toggling the left hand history button. This will show all previous versions of that record
  • Qualifier Configuration - Configuration of qualifiers will now update as they're updated, only showing possible combinations of datasets and attributes
  • Create from Table - Datasets can now be created from a pre-existing table, attributes will be added and the validation will the closest matching one
  • Consistency Checking - The staging tables are now checked for differences. For example, if the attribute 'Type' is a varchar(20) on the staging table, but an int(4) on the audit table, the dataset will not be available for upload until the discrepancy is corrected
  • Invalid Upload Files - Uploaded files that aren't valid CSV or contain invalid encoding (i.e. not UTF-8) now generate error messages indicating the row with the problem
  • Consistent Actions - All action buttons are now located on the left of their respective row which ensures they'll be in a predictable location when the row is of an indeterminate width (i.e. when viewing records)

Bug Fixes

  • Tooltips should displace other visible elements less on older versions of Internet Explorer (Note that this can still be an issue with Internet Explorer 8)

3.6

3.6.11

Bug Fixes

  • Remove excessive database load when processing errors

3.6.9

Bug Fixes

  • Upgrade third-party library due to security issues
  • Qualifier clash checks are now done at the record level, only uploads that contain the exact same key will cause a qualifier overlap

3.6.8

Bug Fixes

  • Error download links are now split by level and constrained, this removes the ability to download an excessive number of errors and tie up workers

3.6.6

Bug Fixes

  • Upgrade third-party library due to security issues

3.6.5

Bug Fixes

  • Update third-party library due to security issues

3.6.4

Bug Fixes

  • Fix display issues with Dataset attribute reordering arrows in Chrome
  • Disable all inputs when changing the Dataset to upload to, this removed ability to start an upload to the wrong dataset
  • Upgrade code highlighter used in expression validations, this now has greater browser support

3.6.3

Bug Fixes

  • Fix undone uploads being considered stale and therefore abandonable
  • Fix mandatory icon not displaying in some situations
  • Upload attempts are now cancelled when the related dataset is removed

3.6.2

Bug Fixes

  • Fix some icons not displaying in IE8
  • Fix javascript errors causing uploads to appear to not progress in IE8
  • Reduced duration of upload screen loading when the user has access to a large number of timed qualifiers
  • Warnings generated during interactive addition of a row no longer disappear once the row is added

New Features

  • Error Download - Download errors in the same format as the current upload this allows easy correction of validation errors
  • Expression Validations - Expression validations now have access to a 'global header', this system preference defines a prefix that is applied to the beginning of all expression validations
  • Viewing Rows - When viewing row data, either as search results or the rows of an upload, the table header will stick to the top of the screen
  • Notification - Associate two access groups so that an external notification system can subscribe to events generated by users of an access group
  • Undo Upload - The last upload attempt for a given dataset may now be undone. This will return the data in the dataset to its state before that upload, keeping the audit history intact
  • Updated Rows - When viewing rows for an upload attempt rows that will result in an update to an existing row are now marked to differentiate them from inserts, this will also be reflected in the upload attempt status page where the number of inserts and updates will be detailed
  • Update Warnings - If a user uploads data that will result in updates to existing records a warning message can be displayed. This message is set as the system preference update_warning_message
  • UI Improvements
    • The attribute validation screen has been reformatted and condensed, only applicable fields are show
    • Viewing and editing a dataset are now one step, users without access to edit will still be shown the informational version
    • The attribute validation list for a given dataset attribute is now visually grouped by type
    • When creating an attribute validation while editing a dataset, once complete you will be returned to the same dataset
    • Dataset attributes are now re-orderable, this order will be reflected in any screen that shows row for example when working with rows for an upload attempt
    • Multi-select lists are now visible as side by side lists so the selected values are more visible, a filter for searching through the list is available as well
    • When editing a dataset, dataset formats, row validations, and aggregate validations now have quick links at the top of their tables
    • Qualifier access can now be defined in one entry instead of having one entry for each value granting access
    • Warnings are now coloured yellow to better distinguish them from higher level errors

Bug Fixes

  • Viewing invalid rows for an upload attempt no longer fails when the error is located in unknown attributes
  • Fixed situation where displayed dataset attributes aren't in a consistent order, attributes that were natural keys were not sorted correctly, they now appear in the same order as defined in the dataset
  • Uploaded files no longer cause an error when the extension is not lower case
  • Generating error summary reports with invalid input now correctly returns an error instead of returning a blank page
  • Fix situation where rows with warnings were considered invalid and would cause Integrity to report that there were 0 rows to be uploaded
  • Exporting search results in a dataset format that has attributes with no position (i.e. virtual attributes), no longer returns an error
  • Dates that lie outside the range 1753..9999 will be marked as invalid in Integrity and are no longer passed through to the database which would cause errors for databases that didn't support large date ranges
  • The begin upload button is no longer disabled in Firefox when pressing back after starting an upload
  • Uploads created by users who have had their access removed no longer get stuck and will self cancel
  • Paginate stale uploads screen to cater for installs with a large number of upload attempts meeting the stale criteria
  • Fix inconsistent behaviour when dates are included as part of a natural key
  • Fix inconsistent behaviour when date columns have a uniqueness constraint
  • Autocomplete input form elements no longer fail when attempting to autocomplete with too many values
  • Changing the type for an attribute validation no longer fails when the underlying column has no explicit limit
  • Reports generated with invalid input now return a warning instead of the general error page
  • Stale uploads will be paginated so large amounts of stale uploads don't stall the interface
  • Users can no longer remove the last group administrator in an access group
  • When creating a dataset from a sample file, validations with ranges no longer cause the creation to fail
  • Removing the length constraint on a String attribute validation no longer fails unexpectedly
  • Qualifier dataset attributes that are reconfigured to not be qualifiers don't cause issues when querying the database
  • Date validated dataset attributes no longer cause unexpected validation errors when part of a natural key
  • HTML is not sent when failing authentication via the API
  • Upload attempts now display quicker after validation when there are a large number of errors
  • Virtual attributes for a dataset format no longer require a column position and are ignored when exporting data in said format
  • CSV dataset formats no longer allow the column delimiter to be blank
  • When abandoning an invalid upload attempt '0 rows abandoned' has been replaced with 'Upload abandoned'
  • Expression validations default to invalid unless explicitly instructed that the row is valid with row_is_valid
  • Dataset attributes that are marked as unique or are part of a natural key now understand typed values are the same, i.e. 01/11/2007 & 1/11/2007 express the same value
  • Fix checking the email on validate checkbox when creating an upload attempt having no effect
  • Users will no longer be shown that they have missing uploads to datasets that they don't have permission to upload to
  • Complex lookup validations now correctly return valid when either side of the lookup is missing
  • Fix situation where temporal caches were not being cleared when an upload attempt was abandoned/completed
  • Fix 'Add missing columns' failing when the dataset's table name contains a dash

3.5

New Features

  • Go Screen - The new upload and home screens have been combined and redesigned to streamline the process of working with uploads in Integrity
  • Automatic Dataset Creation - When creating a new dataset you may now supply a sample CSV file and Integrity will create the dataset for you. The dataset attributes will be based on the header row of the file and the attribute validations will be extracted from the first 10 rows of the sample file
  • Canned Validations - New Integrity instances now have a set of pre-defined attribute validations covering common cases such as email addresses, Australian states, AU/US date formats, etc
  • Cancelling Uploads - Uploads may now be cancelled so they aren't progressed further once any current work is completed
  • Row Error Navigation - Rows are now sortable when viewing all/valid/invalid rows from an upload attempt
  • Keyboard Navigation - When working with rows in an upload attempt the enter key will submit the row
  • Attribute Validation Errors - Errors generated when an attribute validation type check is violated now include the name of the attribute that violated the rule, i.e. Date of Birth is not a valid date
  • Expression Validation Test Attributes - When configuring an expression validation the attributes displayed for the tests is now configurable
  • Upload Attempt Search - Upload attempt search results are now shown most recent first, and the results are orderable
  • Staged Data Export - A dataset's staged data may now be exported as CSV and the export limit has been removed, the limit still applies to XML exports

Bug Fixes

  • Only show one error when a column lookup fails due to an incorrect field value
  • Don't allow string attribute validations to allow a minimum-maximum length range that exceeds the attributes that use it
  • If substitution fails for an uploaded file the upload attempt is cancelled instead of allowing invalid data to continue through to upload
  • Downloading error summaries now have the header row in the correct order, previously the header was output in a fixed order regardless of the format specified
  • Self referential lookup attribute validations are explicitly disallowed when the attribute forms part of a natural key, previously this can cause an infinite loop when performing validation. Attributes may refer to the dataset they are defined in via a lookup validation as long as they are not part of the natural key
  • Downloading row errors of a certain severity level on an upload attempt would download all of the errors regardless of level, each link now downloads the correct subset of errors
  • Inactive expression validations no longer render a blank page when viewed
  • Timed qualifiers could cause the home screen to take 5+ minutes to render when configured with a complex set of rules
  • Fix scenario where users could view upload attempts that their access configuration would not allow them to create
  • Inactive row validations now have action buttons
  • Failed LDAP authentication no longer results in an error page
  • When an upload is queued for staging, the success message now indicates that the upload is queued, not that it has happened
  • Expression validation tests now show more clearly their expected and actual outcomes
  • Importing users from LDAP no longer causes an error when two different distinguished names share the same email attribute
  • Error messages would sometimes display on subsequent pages after navigating away, this is now fixed
  • Fix invalid rows being unsortable after correcting one
  • Fix expression validation tests being unable to be created with an expectation of Invalid
  • Upload attempts no longer freeze when a user loses access to a qualifier or dataset during the upload process
  • Fix issue where correcting rows errors or updating rows with only warnings can cause Integrity to fail when displaying invalid rows and when downloading error summaries
  • Fix situation where rows with warnings were considered invalid and would cause Integrity to report that there were 0 rows to be uploaded

3.4

3.4.12

Bug Fixes

  • Fix issue where correcting rows errors or updating rows with only warnings can cause Integrity to fail when displaying invalid rows and when downloading error summaries
  • Fix situation where rows with warnings were considered invalid and would cause Integrity to report that there were 0 rows to be uploaded

3.4.11

Bug Fixes

  • Fix issue where upload attempts may be failed with message 'Cannot transition status via : begin_uploading from :upload'
  • Fix issue where importing users form LDAP with the same email address as an existing user but a different DN would cause an error, the user is now skipped as having a duplicate email address

3.4.10

New Features

  • Password Reset - Password reset can be disabled for when Integrity authenticates against an external LDAP server

Bug Fixes

  • Fix issue where date validations may be processed incorrectly

3.4.9

Bug Fixes

  • Fix situation where users were unable to query staged data using qualifiers they had access to

3.4.7

New Features

  • Cancelling Uploads - Allow upload attempts to be cancelled while being transferred or validated

New Features

  • Initial Setup - When Integrity is initially installed and there are no users the installer will be prompted to create an administrative user and allows them to batch other users
  • Reload Configuration - Administrators can instruct Integrity to flush any database configuration it may have cached so it may pick up any changes to the database that have happened outside Integrity
  • Create Staging Tables - When creating a dataset Integrity will manage the creation of the staging and audit tables for you. Note that this feature is off by default
  • Add Missing Columns - After adding dataset attributes Integrity can add the related columns to the underlying staging and audit tables for you. Note that this feature is off by default
  • Integrity Upload Performance - Integrity performs its work in smaller more manageable batches so memory usable will not balloon and degrade system performance
  • New Users - The new user email has been replaced and is now clearer
  • Upload Progression - Integrity will no longer ask questions during the upload process if the previous step completed successfully. The upload will now automatically move through validation, aggregate validation, and re-validation steps if a submitted file contains no errors. This means that integrations will need to check with Integrity before attempting to process the upload attempt. See the integration documentation for more information
  • Row Errors - When viewing rows with errors from an upload attempt, the errors are now hyperlinked so focus is moved to the value that is responsible for the error
  • Maintenance - Integrity will now display a maintenance page when work is being carried out during upgrades and planned downtime
  • Stale Uploads - Incomplete upload attempts that have not be worked on in the past month are no considered stale and can now be cancelled by an administrator
  • About - The about page may contain sensitive information and has been removed
  • Expression Validation Syntax - Expression validations will now alert the user that they contain invalid syntax when saved

Bug Fixes

  • Users who had lost permission to replace all data but had previously completed a replace all upload could still start replace all upload attempts
  • Users could previously upload to a dataset with dataset attributes that had no corresponding staging table column, the upload is now stopped and the user warned
  • Range validations now works with dates and can accept negative numbers. See the range validation documentation for more information
  • Integrity file formats now default to automatic for line endings. However if there is multi-line data in the file then we now recommend using the explicit Windows line ending setting for file formats.
  • Internet Explorer users couldn't use the autocomplete function, this is now fixed
  • When creating a multi-lookup row validation inactive datasets were presented as options, these are now hidden
  • When enforcing dataset attribute display and column name uniqueness, previously deleted attributes would be included in the check and the new dataset attribute would not be allowed
  • Fixed upload attempt search failing when the start and end date range values were equal. Equal dates now correctly return upload attempts for that day
  • When displaying dataset attributes (i.e. in drop down lists) the values will now be sorted by display name to improve the visual consistency
  • Data exported as CSV now honours the output dataset formats header row configuration
  • Previously uploaded XML could be invalid however the upload would process with unpredictable results. Integrity will now reject invalid XML files
  • Natural key validation no longer returns incorrect results when the key contains null values or invalid dates
  • Integrity no longer allows dataset attributes to be given column names with non-printing and non-database-friendly characters

3.3

New Features

  • Invalid Encoding - Integrity will disallow an upload attempt with an error if the uploaded file is not UTF-8 compliant
  • Unique Dataset Attributes - Dataset modellers can now enforce a uniqueness check on a single dataset attribute, values for that attributes will be checked for uniqueness across all staged data
  • Missing Dataset Attributes - Integrity will now warn a dataset modeller if a dataset attribute's specified column doesn't exist in the staging tables by highlighting the row yellow
  • Improved Upload Performance - Integrity now caches qualified set information and file content information so validation can be performed faster and with less database interaction

Bug Fixes

  • In instances where Integrity allows in-line editing, it is no longer possible to mis-click on an adjacent row and lose any changes made to the current row
  • Integrity could fail when exporting staged data as CSV when the dataset and the dataset format had different attributes, Integrity now honours the dataset format correctly
  • Qualifier clash detection previously wouldn't consider currently staging uploads as a clash, this will now trigger a revalidation
  • Integrity no longer models money explicitly, existing Money attribute validation will be converted to decimal. See the guidelines for modelling money for more information
  • Internet Explorer 9 is now correctly support

3.2

New Features

  • DB2 Support - DB2 is now a supported database, for both staging and metadata databases
  • Simplified Aggregate Validations - 'Upload prevented until fixed' and 'Exclude from Upload' are not valid actions for invalid aggregate validations as the validation is on the entire data set, not a row or attribute within the dataset
  • Expression Validation Tests - Tests may now be added to expression validations so they can be verified without performing uploads against them
  • Improved Upload Performance - Integrity now performs more caching which will in turn improve validation performance. This introduces a preparation step at the beginning of an upload which will affect integrations as the integration will need to check whether the upload is ready to proceed into validation. See the integration documentation for more information
  • Various User Interface Improvements

Bug Fixes

  • Fix Integrity incorrectly using the system's hostname in generated emails, the notification hostname system preference is now used
  • Fix rows being incorrectly marked as valid when validated against a lookup dataset that was empty
  • Validation errors triggered when uploading XML files are now correctly displayed to the user
  • Users with access to the same dataset through multiple access groups would cause that dataset to be shown twice when creating a new upload, datasets are now shown only once
  • Qualifier names could be displayed multiple times when viewing uploads, they are now only displayed once
  • Fix situation when using MS SQL Server as the staging database, attributes defined with the float data type were not be processed correctly when staging the data into a temporary table
  • Fix Integrity not specifying field sizes when staging into MS SQL Server, the destination table is queried to define the temporary table
  • Minor performance improvement when viewing invalid rows on an upload attempt
  • Qualifiers can now be used when creating tests for expression validations
  • Fix issue where dataset attributes may share display and/or column names, these values must be unique for the given dataset
  • Integrity now handles 0 bye files correctly and will warn the user
  • Multiple errors on the one record are now counted as one error when displaying the error summary.
  • Fix instance where importing an aggregate validation where two datasets have the same attribute name
  • Fix the Select all option on the new upload screen not working when the qualifier has values with spaces in them
  • Information messages are now displayed correctly on the users screen
  • Creating dataset attributes with a column name of audit_id is now prevented, audit_id is a reserved column name
  • It was possible to delete a row, and then edit the row before Integrity had time to process the delete, the row is now removed before the user can edit it
  • Fix scenario where it was possible for users to upload to data sets using qualifiers they do not have access to through search for this data and then clicking on the edit button