[Show all top banners]

pkr_kancha
Replies to this thread:

More by pkr_kancha
What people are reading
Subscribers
:: Subscribe
Back to: Computer/IT Refresh page to view new replies
 DevOps or BigData Hadoop
[VIEWED 10566 TIMES]
SAVE! for ease of future access.
Posted on 01-04-17 7:24 AM     Reply [Subscribe]
Login in to Rate this Post:     0       ?    
 

Hello all,

I'm not sure either I should take DevOps IT course or Hadoop developer. I don't know much about DevOps but I heard it's market is good. On the other hand Hadoop developer seems interesting until you start learning Map reduce (programming). What do you guys recommend to get involved with? Does anyone here has experience on either DevOps or Hadoop? Honest opinions would be appreciated.
 
Posted on 01-04-17 1:27 PM     [Snapshot: 141]     Reply [Subscribe]
Login in to Rate this Post:     2       ?     Liked by
 

I think it soley depends on your ability and interest on those domains. They are completely different fields of IT. I do somewhat similar to DeVOps per say . I would think that DevOps is a fancy name for system admin for products usually in cloud or in virtual environments. If you have general IT experience and good understanding of software workflow, DevOps should be fairly easy to pick up. DevOps can be fascinating and incredibly challenging at the same time ...so many tools and technology to learn from.

Hadoop on the other hand I hear is complex right from the start. The things it can do is complex is used in conjunction with other complex products and to be good at hadoop you have to be good at programming ...really good. That's my thought after installing and playing with hadopp / yarn etcs.
That being said I have seen people just push jobs on hadoop for a living and make 6 figures so LUCK I guess is also an important factor.
Hadoop is fairly new and in my opinion will be replaced with some new stuff soon like for eg spark or bigquery. Mapreduce is just the tip of the iceberg...
I think of Hadoop as an ecosystem more than a technology...database / cluster / spark / storm / pig ...etc all falls under that ecosystem.


Good luck


 
Posted on 01-04-17 1:58 PM     [Snapshot: 158]     Reply [Subscribe]
Login in to Rate this Post:     0       ?    
 

fdpower!! Thank you for the great breakdown.

Your explanation regarding Hadoop is spot on. Its complex from day 1. While I have found HDFS, PIG, HIVE fairly learnable. Map Reduce is where I find it complex. I've spend few hours to analyze the code in one program and I bet it's not even that complex for people who have strong Java background. Lots of people in forums and reddit are saying that they don't even use Map Reduce anymore since tools like HIVE, PIG have come up as an alternative to Map Reduce. But I guess you have to master your Map Reduce skills in case those alternatives are not available when you go to a new company.

I have fairly a good background in IT including Linux, Networking. If I go for DevOps would it be a good idea?
 
Posted on 01-05-17 10:11 AM     [Snapshot: 346]     Reply [Subscribe]
Login in to Rate this Post:     0       ?    
 

Hi Bro...see the requirement of Hadoop .... what you want to be in hadoop .. admin or dev... admin part is easy as you have knowledge of Linux and hadoop is open source and working on Linux... like windows it has its own file system like HDFS.... I know hadoop admin part that is pretty easy... but if you want to be dev then you must have a programming ability. with hadoop lots of similar third party tool used in it... Java is good if you want to for dev... that give you a basic to understand all ...

Good luck....if you want to any other technology you can msg me...
 
Posted on 01-05-17 8:40 PM     [Snapshot: 431]     Reply [Subscribe]
Login in to Rate this Post:     1       ?     Liked by
 

In addition to fdpower: If you want to be a Hadoop developer, you need to know the commonly used tools/components in hadoop ecosystem. It can be fairly simple or complex at the same time to be a hadoop/big data developer and depends on which side of the hadoop ecosystem you are comfortable working with.

Consider Hadoop as a black box and lets say there is a left side of the box(1); the box itself(2); there is a right side of the box(3) & there is also a enclosure for the black box(4).

Left Side of the box(1): This is often termed as big data ingestion in the world of hadoop. The tools like Apache Sqoop, Kafka, Flume, Flafka, HDF (Hortonworks)..etc are used.

Box itself(2): Nowadays there are various open source tools like (Hive, Pig..etc) that are wrapper around old/gold MapReduce; you don't really need to write core MR code in java to execute your job. You should consider learning about the basic architecture of Hadoop, Yarn, Zookeeper, Mappers, Reducers, Partitioners, RecordReader, RecordWriter, job, job tracker etc... and also some knowledge of core java.

Right Side of the box(3): This is where you apply your business logic; Big Data analytics or Big Data Scientists, a sort of fancy name in Hadoop world. These guys treat Hadoop as a distributed storage unit. Pig & Hive are widely used for batch processing; analytics tools like Spark, HBase, Storm, Phoenix (SQL wrapper for HBase) & reporting tools like PentaHo, SAS.. etc are widely used. If you are a Phd in Physics, mathematics, even Biology, Chemistry.. you will be working in this side of Hadoop; extracting data from hadoop bridge (iPhython Notebook) or java and implement your complex machine learning, image analysis, etc.

Box Enclosure(4): This is where Dev Ops or Sys Admin role comes into play. Companies prefer preconfigured Hadoop cluster from either Cloudera or Hortonworks.. and buy subskriptions for support from these guys. Cloud service like AWS, Microsoft Azure, Rackspace..etc also provide excellent support for Cloudera or Hortonworks.

p.s. it turned out to be a long post. This is a little insight from a person who has background in Hadoop/Big Data; I hope it helps.
 
Posted on 01-05-17 11:38 PM     [Snapshot: 497]     Reply [Subscribe]
Login in to Rate this Post:     0       ?    
 

aaisunkaata !! Thanks for the feedback. Really appreciated.

Ruben123!! Very nice and detailed information regarding Hadoop/BigData. I really appreciate your isight.

I've been learning Hadoop for about a month now and all the materials EXCEPT writing MR code has been interesting and fun ride. I hope MR will be a smooth ride as I progress into the world of Big Data.


 
Posted on 02-05-17 10:16 AM     [Snapshot: 770]     Reply [Subscribe]
Login in to Rate this Post:     0       ?    
 

Can you suggest any consultancy who does placemnet for hadoop. ?
 
Posted on 02-05-17 12:50 PM     [Snapshot: 829]     Reply [Subscribe]
Login in to Rate this Post:     0       ?    
 

I haven't found one either. Please someone suggest if they know a consultancy that does placements in Hadoop.
 
Posted on 02-05-17 2:52 PM     [Snapshot: 868]     Reply [Subscribe]
Login in to Rate this Post:     0       ?    
 

interesting..
ruben123, well explained...
Can you plz fairly explain providing a real time illustration how Big data implementation benefits?
i know, It could be used everywhere..,social network, stocks, politics, hospital etc
Back in 2012, I did a POF on BigData using Horton works. The project used python skript for analyzing chunk of social network data.
Right now, we are implementing it in Stock.
 
Posted on 02-05-17 11:56 PM     [Snapshot: 976]     Reply [Subscribe]
Login in to Rate this Post:     0       ?    
 

Last edited: 10-Feb-17 08:03 PM

 


Please Log in! to be able to reply! If you don't have a login, please register here.

YOU CAN ALSO



IN ORDER TO POST!




Within last 365 days
Recommended Popular Threads Controvertial Threads
Lets play Antakshari...........
डीभी परेन भने खुसि हुनु होस् ! अमेरिकामाधेरै का श्रीमती अर्कैसँग पोइला गएका छन् !
शीर्षक जे पनि हुन सक्छ।
What are your first memories of when Nepal Television Began?
[New post] Why Would Krishna Have To Run From The Battlefield
Sajha Poll: नेपालका सबैभन्दा आकर्षक महिला को हुन्?
ChatSansar.com Naya Nepal Chat
is Rato Bangala school cheating?
Why always Miss Nepal winner is Newari??
NRN card pros and cons?
Basnet or Basnyat ??
पुलिसनी संग - आज शनिवार - अन्तिम भाग
निगुरो थाहा छ ??
Nas and The Bokas: Coming to a Night Club near you
श्राद्द
सेक्सी कविता - पार्ट २
TPS Re-registration
What Happened to Dual Citizenship Bill
पाप न साप घोप्टो पारि थाप !!
अमेरिकामा छोरा हराएको सूचना
Nas and The Bokas: Coming to a Night Club near you
राजदरबार हत्या काण्ड बारे....
Mr. Dipak Gyawali-ji Talk is Cheap. US sends $ 200 million to Nepal every year.
Harvard Nepali Students Association Blame Israel for hamas terrorist attacks
TPS Update : Jajarkot earthquake
is Rato Bangala school cheating?
NOTE: The opinions here represent the opinions of the individual posters, and not of Sajha.com. It is not possible for sajha.com to monitor all the postings, since sajha.com merely seeks to provide a cyber location for discussing ideas and concerns related to Nepal and the Nepalis. Please send an email to admin@sajha.com using a valid email address if you want any posting to be considered for deletion. Your request will be handled on a one to one basis. Sajha.com is a service please don't abuse it. - Thanks.

Sajha.com Privacy Policy

Like us in Facebook!

↑ Back to Top
free counters