Automated Data Validation Framework for Data Quality in Big Data Migration Projects

International Journal of Computer Science and Engineering

© 2014 by SSRG - IJCSE Journal

Volume 1 Issue 10

Year of Publication : 2014

Authors : V. Rathika,, Dr. L. Arcokiam

10.14445/23488387/IJCSE-V1I10P105

How to Cite?

V. Rathika,, Dr. L. Arcokiam, "Automated Data Validation Framework for Data Quality in Big Data Migration Projects," SSRG International Journal of Computer Science and Engineering , vol. 1,  no. 10, pp. 1-5, 2014. Crossref, https://doi.org/10.14445/23488387/IJCSE-V1I10P105

Abstract:

The process of moving vast amount of data from one place to another is called big data migration. Huge volume of data is extracted, transformed, structured and loaded from legacy data base into a newer structure in the process which leads to data corruption. Data validation testing is essential after data migration process over to ensure data quality. To perform efficient data migration process, source data are mapped to new system which handles all the data formats. It is important when upgrade and relocation of existing systems. Businesses are creating significant data management challenges by increasing volumes of data. They should be able to access and organize volumes of data stored in a variety formats. Compare to manual process, automate data validation improve data quality in less time and cost with good quality. This paper emphasis on proposing model to do automatic quality checks for huge volume of data migrations.

Keywords:

Architecture, Data Migration, Data Quality, Framework, Validation Testing.

References:

[1] Shinde Anita Vitthal, Thite Vaishali Beban, Roshini Warade and Krupali Chaudhari.: Data Migration System in Heterogeneous Database, in International Journal of Engineering Science and Innovative Technology, 2(2), pp. 88–92, (March 2013). 
[2] Klaus Haller.: Towards the Industrialization of Data Migration: Concepts and Patterns for Standard Software Implementation Projects, in Springer, pp. 63–78, (2009). 
[3] Vishnu B, Manjunath T N and Hamsa C.: An Effective Data Warehouse Security Framework, in International Journal of Computer Applications Recent Advances in Information Technology, pp. 33–37, (2014). 
[4] Manjunath T N, Ravindra S Hegadi and Archana R A.: A Study on Sampling Techniques for Data Testing, in International Journal of Computer Science and Communication, 2(1), pp. 13–16, (June 2012). 
[5] Florian Mathhes, Christopher Schulz and Klaus Haller.: Testing & Quality Assurance in Data Migration Projects, In 27th IEEE International Conference on Software Maintenance, Williamsburg, pp. 25–30, (2011). 
[6] Manjunath T N, Ravindra S Hegadi and Mohan H S.: Automated Data Validation for Migration Security, in International Journal of Computer Applications, 30(6), pp. 41–46, (September 2011). 
[7] Priyanka Paygude, Devale P R.: Automated Data Validation Testing Tool for Data Migration Quality Assurance, in International Journal of Modern Engineering Research, 3(1), pp. 599–603, (February 2013). 
[8] Manjunath T N, Ravindra S Hegadi and Ravikumar G K.: Analysis of Data Quality Aspects in Data Warehouse Systems, in International Journal of Computer Science and Information technologies, 2(1), pp. 477–485, (2011). 
[9] Priyanka Paygude, Devale P R.: Automation of Data Validation Testing for QA in the Project of DB Migration, in International Journal of Computer Science Engineering and Information Technology Research, 3(3), pp. 15–22, (August 2013). 
[10] Ranjith Singh, Kawaljeet Singh.: A Descriptive Classification of Causes of Data Quality Problems in Data Warehousing, in International Journal of Computer Science Issues, 7(3), pp. 45–51, (May 2010). 
[11] Haller K.: Towards the Industrialization of Data Migration: Concepts, in 21st International Conference on Advanced Information Systems Engineering, Netherland, pp. 70–78, (2009). 
[12] Atsa Etoundi Roger, Abessolo Alo’o Ghisiain and Simo Bonaventure Joel.: Migration of Legacy Information System based on Business Process Theory, in International Journal of Computer Applications, 33(2), pp. 27–34, (November 2011).