Google scholar arxiv informatics ads IJAIS publications are indexed with Google Scholar, NASA ADS, Informatics et. al.

Call for Paper

-

April Edition 2021

International Journal of Applied Information Systems solicits high quality original research papers for the April 2021 Edition of the journal. The last date of research paper submission is March 15, 2021.

Performance Overhead on Relational Join in Hadoop using Hive/Pig/Streaming - A Comparative Analysis

Prabin R. Sahoo Published in Information Sciences

International Journal of Applied Information Systems
Year of Publication: 2012
© 2012 by IJAIS Journal
Authors Prabin R. Sahoo
10.5120/ijais12-450799
Download full text
  1. Prabin R Sahoo. Article: Performance Overhead on Relational Join in Hadoop using Hive/Pig/Streaming - A Comparative Analysis. International Journal of Applied Information Systems 4(7):15-20, December 2012. BibTeX

    @article{key:article,
    	author = "Prabin R. Sahoo",
    	title = "Article: Performance Overhead on Relational Join in Hadoop using Hive/Pig/Streaming - A Comparative Analysis",
    	journal = "International Journal of Applied Information Systems",
    	year = 2012,
    	volume = 4,
    	number = 7,
    	pages = "15-20",
    	month = "December",
    	note = "Published by Foundation of Computer Science, New York, USA"
    }
    

Abstract

Hadoop Distributed File System (HDFS) is quite popular in the big data world. It not only provides a framework for storing data in a distributed environment, but also has set of tools to retrieve and process these data using map-reduce concept. This paper discusses the result of evaluation of major tools such as Hive, Pigand hadoop streaming for solving problems from a relational prospective and comparing their performances. Though big data cannot be compared to the strength of relational database in solving relational problems, but as big data is about data so the relational nature of data access cannot be eliminated altogether. Fortunately, there are ways to deal with this which has been discussed in this paper from a performance prospective. This may help the big data community in understanding the performance challenges so that further optimization can be done and the application developers' community can learn how strategically the relational operations need to be used.

Reference

  1. Lucene Hadoop, "Hadoop Map-Reduce Tutorial", http://hadoop. apache. org/docs/r0. 15. 2/mapred_tutorial. html, retrieved online November 2012
  2. Viglas,S. D,Niazi,S, "SAND Join — A skew handling join algorithm for Google's Map/Reduce framework", http://ieeexplore. ieee. org/xpl/articleDetails. jsp?tp=&arnumber=6151466&contentType=Conference+Publications&queryText%3Djoin+in+hadoop, Dec 2011
  3. Ashish Thusoo, Joydeep Sen Sarma, Namit Jain, Zheng Shao, Prasad Chakka, Ning Zhang, Suresh Antony, Hao Liu, and Raghotham Murthy, "Hive – A Petabyte Scale Data Warehouse Using Hadoop", infolab. stanford. edu/~ragho/Hive-icde2010. pdf,ICDE 2010
  4. Christer A. Hansen, "Optimizing Hadoop for the cluster", Institue for Computer Science, University of Troms0, Norway, http://oss. csie. fju. edu. tw/~tzu98/Optimizing%20Hadoop%20for%20the%20cluster. pdf, Retrieved online October 2012
  5. Nils Braden, "The Hadoop Framework", http://homepages. thm. de/~hg51/Veranstaltungen/MasterSeminar1011/NielsBraden. pdf, Retrieved September 2012
  6. Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler, The Hadoop Distributed File System, http://storageconference. org/2010/Papers/MSST/Shvachko. pdf, retrieved online October, 2012
  7. Apache Hadoop,"Hadoop 0. 20 documentation", http://hadoop. apache. org/docs/r0. 17. 1/mapred_tutorial. html, August 2008, retrieved online August 2012
  8. Atlassian Confluence, "Hive Tutorial", https://cwiki. apache. org/Hive/tutorial. html, Feb 2011
  9. Apache Hadoop, "Pig 0. 7. 0 Documentation", http://Pig. apache. org/docs/r0. 7. 0/tutorial. html, retrieved online August, 2012
  10. HDFS Architecture Guide, http://hadoop. apache. org/docs/hdfs/current/hdfs_design. html, retrieved online October, 2012
  11. Map/Reduce Tutorial, http://hadoop. apache. org/docs/r0. 20. 2/mapred_tutorial. html, retrieved online October, 2012

Keywords

Hive, Pig, Hadoop, HDFS, Map-Reduce, streaming