Documentation
Related Projects
Welcome to Apache& Hadoop&!
What Is Apache Hadoop?
The Apache& Hadoop& project develops open-source software for
reliable, scalable, distributed computing.
The Apache Hadoop software library is a framework that
allows for the distributed processing of large data sets across clusters
of computers using simple programming models. It is designed to scale up
from single servers to thousands of machines, each offering local
computation and storage. Rather than rely on hardware to deliver
high-availability, the library itself is designed to detect and handle
failures at the application layer, so delivering a highly-available
service on top of a cluster of computers, each of which may be prone to
The project includes these modules:
Hadoop Common:
The common utilities that support the other Hadoop modules.
Hadoop Distributed File System (HDFS&):
A distributed file system that provides high-throughput access to application data.
Hadoop YARN:
A framework for job scheduling and cluster resource management.
Hadoop MapReduce:
A YARN-based system for parallel processing of large data sets.
Other Hadoop-related projects at Apache include:
A web-based tool for provisioning, managing, and monitoring Apache Hadoop
clusters which includes support for Hadoop HDFS, Hadoop MapReduce, Hive,
HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides
a dashboard for viewing cluster health such as heatmaps and ability to
view MapReduce, Pig and Hive applications visually alongwith features to
diagnose their performance characteristics in a user-friendly manner.
A data serialization system.
A scalable multi-master database with no single points of failure.
A data collection system for managing large distributed systems.
A scalable, distributed database that supports structured data storage for large tables.
A data warehouse infrastructure that provides data summarization and ad hoc querying.
A Scalable machine learning and data mining library.
A high-level data-flow language and execution framework for parallel computation.
A fast and general compute engine for Hadoop data. Spark provides a simple and
expressive programming model that supports a wide range of applications,
including ETL, machine learning, stream processing, and graph computation.
A generalized data-flow programming framework, built on Hadoop YARN,
which provides a powerful and flexible engine to execute an arbitrary
DAG of tasks to process data for both batch and interactive use-cases.
Tez is being adopted by Hive&, Pig& and other frameworks in
the Hadoop ecosystem, and also by other commercial software (e.g. ETL tools),
to replace Hadoop& MapReduce as the underlying execution engine.
A high-performance coordination service for distributed applications.
Getting Started
To get started, begin here:
reading the documentation.
Hadoop from the
release page.
Hadoop on the
mailing list.
Download Hadoop
Please head to the
page to download a release of Apache Hadoop.
Who Uses Hadoop?
A wide variety of companies and organizations use Hadoop for both research and production.
Users are encouraged to add themselves to the Hadoop
wiki page.
08 October, 2016: Release 2.6.5 available
A point release for the 2.6 line.
Please see the
for the list of 79 critical bug
fixes and since the previous release 2.6.4.
03 September, 2016: Release 3.0.0-alpha1 available
This is the first alpha in a series of planned alphas and betas leading up to a 3.0.0 GA release.
The intention is to "release early, release often" to quickly iterate on feedback collected from downstream users.
Please note that alpha releases come with no guarantees of quality or API stability, and are not intended for
production use.
Users are encouraged to read the
coming in 3.0.0.
detail all the changes since the previous minor release 2.7.0.
25 August, 2016: Release 2.7.3 available
A point release for the 2.7 line.
Please see the
for the list of 221 bug fixes and
patches since the previous release 2.7.2.
11 February, 2016: Release 2.6.4 available
A point release for the 2.6 line.
Please see the
for the list of 46 critical bug
fixes and since the previous release 2.6.3.
25 January, 2016: Release 2.7.2 (stable) available
A point release for the 2.7 line.
Please see the
for the list of 155 bug fixes and
patches since the previous release 2.7.1.
17 December, 2015: Release 2.6.3 available
A point release for the 2.6 line.
Please see the
for the list of 35 critical bug
fixes and since the previous release 2.6.2.
28 October, 2015: Release 2.6.2 available
A point release for the 2.6 line.
Please see the
for the list of 15 critical bug
fixes and since the previous release 2.6.1.
23 September, 2015: Release 2.6.1 available
A point release for the 2.6 line.
Please see the
for the list of 158 critical bug
fixes and since the previous release 2.6.0.
06 July, 2015: Release 2.7.1 (stable) available
A point release for the 2.7 line. This release is now considered
Please see the
for the list of 131 bug fixes and
patches since the previous release 2.7.0. Please look at the
2.7.0 section below for the list of enhancements enabled by this
first stable release of 2.7.x.
21 April 2015: Release 2.7.0 available
Apache Hadoop 2.7.0 contains a number of significant
enhancements. A few of them are noted below.
IMPORTANT notes
This release drops support for JDK6 runtime and works with
JDK 7+ only.
This release is not yet ready for production use. Critical
issues are
being ironed out via testing and downstream
adoption. Production users should wait for a 2.7.1/2.7.2
Hadoop Common
Support Windows Azure Storage - Blob as a file
system in Hadoop.
Hadoop HDFS
Support for file truncate
Support for quotas per storage type
Support for files with variable-length blocks
Hadoop YARN
Make YARN authorization pluggable
Automatic shared, global caching of YARN localized resources
Hadoop MapReduce
Ability to limit running Map/Reduce tasks of a job
Speed up FileOutputCommitter for very large jobs with many
output files.
Full information about this milestone release is available at
18 November, 2014: release 2.6.0 available
Apache Hadoop 2.6.0 contains a number of significant
enhancements such as:
Hadoop Common
Key management server (beta)
Credential provider (beta)
Hadoop HDFS
Heterogeneous Storage Tiers - Phase 2
Application APIs for heterogeneous storage
SSD storage tier
Memory as a storage tier (beta)
Support for Archival Storage
Transparent data at rest encryption (beta)
Operating secure DataNode without requiring root
Hot swap drive: support add/remove data node volumes
without restarting data node (beta)
AES support for faster wire encryption
Hadoop YARN
Support for long running services in YARN
Service Registry for applications
Support for rolling upgrades
Work-preserving restarts of ResourceManager
Container-preserving restart of NodeManager
Support node labels during scheduling
Support for time-based resource reservations in
Capacity Scheduler (beta)
Global, shared cache for application artifacts (beta)
Support running of applications natively in
Docker containers (alpha)
Full information about this milestone release is available at
19 November, 2014: release 2.5.2 available
Full information about this milestone release is available at
12 September, 2014: release 2.5.1 available
Full information about this milestone release is available at
11 August, 2014: release 2.5.0 available
Full information about this milestone release is available at
30 June, 2014: release 2.4.1 available
Full information about this milestone release is available at
27 June, 2014: release 0.23.11 available
Full information about this milestone release is available at
07 April, 2014: release 2.4.0 available
Full information about this milestone release is available at
20 February, 2014: release 2.3.0 available
Full information about this milestone release is available at
11 December, 2013: release 0.23.10 available
Full information about this milestone release is available at
15 October, 2013: release 2.2.0 available
Apache Hadoop 2.x reaches GA milestone!
Full information about this milestone release is available at
25 August, 2013: release 2.1.0-beta available
Apache Hadoop 2.x reaches beta milestone!
Full information about this milestone release is available at
27 December, 2011: release 1.0.0 available
Hadoop reaches 1.0.0!
Full information about this milestone release is available at
March 2011 - Apache Hadoop takes top prize at Media Guardian Innovation Awards
Described by the judging panel as a "Swiss army knife of the 21st century",
Apache Hadoop picked up the innovator of the year award for having the
potential to change the face of media innovations.
January 2011 - ZooKeeper Graduates
Hadoop's ZooKeeper subproject has graduated to become a
top-level Apache project.
Apache ZooKeeper can now be found
September 2010 - Hive and Pig Graduate
Hadoop's Hive and Pig subprojects have graduated to become
top-level Apache projects.
Apache Hive can now be found
Pig can now be found
May 2010 - Avro and HBase Graduate
Hadoop's Avro and HBase subprojects have graduated to become
top-level Apache projects.
Apache Avro can now be found
Apache HBase can now be found
July 2009 - New Hadoop Subprojects
Hadoop is getting bigger!
Hadoop Core is renamed Hadoop Common.
MapReduce and the Hadoop Distributed File System (HDFS) are now separate subprojects.
Avro and Chukwa are new Hadoop subprojects.
See the summary descriptions for all subprojects above. Visit the individual sites for more detailed information.
March 2009 - ApacheCon EU
In case you missed it....
November 2008 - ApacheCon US
In case you missed it....
July 2008 - Hadoop Wins Terabyte Sort Benchmark
One of Yahoo's Hadoop clusters sorted 1 terabyte of data in 209 seconds, which beat the previous record of 297 seconds in the annual general purpose
(Daytona) . This is the first time that either a Java or an open source program has won.Round C APAC Test 2016
题目链接：/codejam/contest/4284487Problem A. gRanksThere are many great athletes in the world, and it's very hard to say who is the best in the world at a particular sport, especially when different athletes have won different competitions. Here's one possible system for ranking athletes:1. Determine the number P of finishing places in any competition that will be worth points for athletes, and how many points Si each of those finishing places is worth. For example, for P = 3, one possible assignment would be 1000 points for 1st place, 500 for 2nd place, and 300 for 3rd place, and 0 for anything below that. (We assume there are no ties within competitions.)2. Since not all competitions are equally important, assign a weight Wi to each one. The score gained by an athlete for a competition will be the points from step 1, modified by the weight for that competition. For example, we may decide that Olympics has a weight of 5, and, continuing with our example from above, the winner of the Olympics would receive 5 * 1000 = 5000 points.3. Since we don't want to reward athletes simply for participating in many competitions, we count only the M highest scores received by an athlete across all competitions. For example, if M = 2 and an athlete earns scores of 0*1, and 300*3 in three different competitions, only the 5000 and 900 would be counted.You are given the points per finishing place, the weights of the competitions, and the results of the competitions. Can you rank all of the athletes who appeared in the competitions? If multiple athletes have the same score, they will share the same rank and listed in alphabetical order of their names.InputThe first line of the input gives the number of test cases, T. T each test case consists of the following:1. One line with an integer P, the number of top places for which points are awarded.2. One line consists with P integers representing the scores Si for the top places, starting with first place and continuing down the places.3. One line with an integer N, the number of competitions. 4. N lines, each of which represents a competition. Each line begins with Wi, the weight of this competition, and continues with the P names of the athletes who finished in the top P places. They are listed in descending order starting from first place.5. One line with an integer M, the maximum number of competitions counted toward an athlete's score.OutputFor each test case, output one line containing &Case #x:&, where x is the test case number (starting from 1). Then output one line for each athlete, from highest rank to lowest rank, with the format r: name, where r is the rank of the athlete and name is the name of the athlete. You need to rank all of the athletes that appeared in the input.Limits题意：给了一个评分系统，求运动员的排名。P表示每场比赛前多少名次有积分，然后给出具体积分。接下来n行表示n个比赛，每行第一个数是这个比赛的权重，之后从第一名开始给出运动员的名字。最后给一个数m表示取前m高的积分（防止过多参加比赛赚取积分）。思路：因为数据范围较小，所以其实给每个运动员一个数据记录积分，之后排序取前m即可。这里用了堆其实是不必要的。#include &cstdio&#include &string&#include &queue&#include &unordered_map&#include &algorithm&#include &iostream&#define N 1005int T,n,m,w,k;int s[N],len = 0;unordered_map&string, int&//运动员名字到对应的堆的映射struct node{}p[N*10];int cmp(node a,node b){
if(a.v == b.v)
return a.s & b.s;
return a.v & b.v;}int main(){
for(int z = 1;z&=T;z++){
priority_queue&int& q[N*10];
hh.clear();
for(int i = 0;i&m;i++)
cin && s[i];
for(int i = 0;i&n;i++){
for(int j = 0;j&m;j++){
if(hh.find(name) == hh.end())
hh[name] = len++;
q[hh[name]].push(w*s[j]);
unordered_map&string, int& ::
for(it = hh.begin();it!=hh.end();++it){
p[len].s = it-&
p[len].v = 0;
int tmp = it-&
for(int i = 0;!q[tmp].empty() && i&i++){
p[len].v += q[tmp].top();
q[tmp].pop();
sort(p, p+len, cmp);
cout && &Case #&&&z&&&:&&&
for(int i = 0;i&){
cout&&i+1&&&: &&&p[i].s&&
int j = i+1;
while(j&len && p[j].v == p[j-1].v){
cout&&i+1&&&: &&&p[j].s&&
return 0;}Problem B. gFilesAlien tech company G has a very old file transfer tool that is still in use today. While the tool is running, it reassures users by giving status updates on both the percentage of files transferred so far and the number of files transferred so far. The status updates during the process might look like this:20% |==&-------| 1 files transferred100% |==========| 5 files transferredBut the percentage isn' it is simply truncated before the decimal point (i.e. floored to the next lowest or equal 1%). That is, both 1.2% and 1.7% would be displayed as 1%.Some users may want to know the exact total number of files, so you want to modify the tool so that the status updates look like this:20% |==&-------| 1 out of 5 files transferred100% |==========| 5 out of 5 files transferredBut you've realized that it may or may not be possible to know the number of files. Given the status updates that the tool displays, either figure out how many files there are, or determine that it can't be done (i.e., there are multiple possible values for the number of files). The status updates are not guaranteed to occur at regular intervals and are not guaranteed to occur whenever a file is transferred.InputThe first line contains T, the number of test cases. T test cases follow. Each test case consists of one line with an integer N, the number of status updates output by the tool, followed by N lines with the format Pi Ki, where Pi and Ki are integers representing the percentage and number of files transferred at some point in the process. The updates are given listed in chronological order -- that is, both the Pi values and the Ki values are nondecreasing throughout a test case.OutputFor each case, output a line starts with &Case #x: y&, where x is the number of the case (starting from 1), and y is either the total number of files, or -1 if that number is ambiguous.题意：给定一个百分比和已经传输的文件数量，问能否准确猜出一共有多少文件，如果结果有歧义输出-1.思路：二分，如果没有合理方案，必然输出-1；如果有，那么看看该方案减一和加一是否合理，如果有合理的，那么输出-1，否则输出方案即可。因为不可能出现两个合理方案中间夹着不合理的情况。特别注意如果方案小于输入中的最大文件数必然是错误的（体现在二分时候的下界，和test函数一开始的return）#include &cstdio&#include &algorithm&#define N 105long long s[N][2];int T,n;int test(long long x){
if(x&s[n-1][1])
return -1;
for(int i = 0;i&n;i++){
long long tmp = s[i][1]*100/x;
if(tmp & s[i][0])
return -1;
if(tmp & s[i][0])
return 0;}long long solve(long long low,long long high){
while(low &= high){
mid = (low+high)&&1;
int j = test(mid);
high = mid-1;
else if(j&0)
low = mid+1;
return -1;}int main(){
scanf(&%d&,&T);
for(int z = 1;z&=T;z++){
printf(&Case #%d: &,z);
long long flag = 0;
scanf(&%d&,&n);
for(int i = 0;i&n;i++)
scanf(&%lld %lld&,&s[i][0],&s[i][1]);
flag = solve(s[n-1][1], 0000);
if(flag != -1){
if(test(flag-1) && test(flag+1))
printf(&%lld/n&,flag);
printf(&-1/n&);
printf(&-1/n&);
return 0;Problem C. gGamesThe country of elves is planning to hold an elimination tournament, and there are 2N elves who would like to take part. At the start of the tournament, they will be given unique ID numbers from 1 to 2N, and the Elf President will line them up in some order.The tournament is a series of matches between two elves, and every match has one winner and one loser (there are no ties). In the first round, the first elf in the line competes against the second elf in the line, the third elf competes against the fourth elf, and so on. After the first round, the 2N-1 elves who lost leave the line, and the 2N-1 elves who won remain where they are. Then, the remaining elves play the second round in the same way: the first remaining elf in the line competes against the second remaining elf in the line, the third remaining elf competes against the fourth remaining elf, and so on. After N rounds, there will be only one elf remaining, and that elf is the winner.M of the elves are sensitive, which means that they will be very sad if they have to compete in matches against their friends during the games. Specifically, the ith elf will be sad if they have to compete with their friends in the first Ki rounds. (Note that friendship is not necessarily mutual: if one elf considers another elf to be a friend, the other elf does not necessarily consider that elf to be a friend.)The Elf President wants to know: is there a way to specify the initial positions of all 2N elves to guarantee that no elf will be sad, no matter what happens in the tournament?InputThe first line of the input gives the number of test cases, T. T test cases follow. Each test case consists of one line with two integers N and M, then M sets of two lines each, in which the first line has integers Ei, Ki, and Bi for one elf, and the second has Bi integer ID numbers of that elf's friends.OutputFor each test case, output one line containing &Case #x: &, where x is the case number (starting from 1), followed by YES or NO.Limits1 ≤ T ≤ 200.0 ≤ M ≤ 2N.1 ≤ Ei ≤ 2N.1 ≤ Ki ≤ N.M ≤ sum(Bi) ≤ min(2 * M, 2N).Small dataset1 ≤ N ≤ 3.Large datasetN = 4.题意：现在想把2^n个人（n&=4)排成一排进行锦标赛。给出若干个限制，比如a和b不能在第1轮相遇等等。问符合限制的排列是否存在。思路：一看数据觉得是搜索加剪枝，但是手写了一个发现剪枝的效果不明显。后来知道是用状态压缩dp来做。首先用0~2^16之间的数字表示一个分配，下标就表示人，对应数字为1的下标表示放在一起，dp值为1表示可行，为0表示不可行（注意1的数量必然是2的幂次）。比如dp[5]表示把第1个人和第3个人放在一起（即第一轮相遇），而dp[27]表示将第1、2、4、5个人放在一个区域。以dp[27]为例，先检查两两之间在第二轮相遇是否有限制，如果有某两个不允许相遇，那么直接返回dp[27]=0；否则枚举平分递归判断下去。#include &cstdio&#include &cstring&#include &vector&#include &cstdlib&#define N (1&&n)int T;int n,m;int g[20][20];int dp[1&&16];vector&int& flag[4];int dfs(int x, int d){
if(dp[x] != -1)
return dp[x];
return dp[x] = 1;
vector&int&
for (int i = 0; i&N; ++i)
if ((1&&i) & x)
hh.push_back(i);
for (int i = 0; i & hh.size(); ++i)
for (int j = i+1; j & hh.size(); ++j)
if (g[hh[i]][hh[j]] &= d)
return dp[x] = 0;
auto len = flag[d].size();
for(int i = 0;i&i++){
int a = 0;
for(int j = 0;j&(1&&d);++j)
if((1&&j)&flag[d][i])
a |= 1&&hh[j];
int b = x-a;
if(a&b && dfs(a, d-1) && dfs(b, d-1))
return dp[x] = 1;
return dp[x] = 0;
}int onenum(int x){
int res = 0;
while (x) {
x &= (x-1);
}}int main(){
for(int i = 1;i&=4;i++)//flag[i]表示从2^i中选择2^(i-1)个数的方案
for(int j = 1;j&(1&&(1&&i));j++){
int tmp = onenum(j);
if(tmp == 1&&(i-1))
flag[i].push_back(j);
scanf(&%d&,&T);
for(int z = 1;z&=T;z++){
int a,b,k,
scanf(&%d %d&,&n,&m);
memset(g, -1, sizeof(g));
memset(dp, -1, sizeof(dp));
for(int i = 0;i&m;i++){
scanf(&%d %d %d&,&a,&k,&num);
while(num--){
scanf(&%d&,&b);
g[a-1][b-1] = g[b-1][a-1] = max(g[a-1][b-1], k);
if(dfs((1&&N)-1, n))
printf(&Case #%d: YES/n&,z);
printf(&Case #%d: NO/n&,z);
return 0;}Problem D. gMatrixYou have a square N by N matrix M of nonnegative integers. We would like to make a list of the maximum values in every sub-matrix of size K by K within M, and then find the sum of those values together. (Note that the same entry of M might be the maximum value in more than one sub-matrix, in which case it will show up multiple times in the list.) Can you find that sum?To simplify the input of the matrix, you are given two arrays A and B of length N, and two integers C and X. Then the entry Mij (for the ith row and jth column of the matrix) equals (Ai*i+Bj*j + C) mod X, where i and j are in the range [1, N].InputThe first line of the input gives the number of test cases, T. T test cases follow. Each test case begins with one line with four integers, N, K, C and X. Then there are two lines with N integers each, representing the arrays A and B.OutputFor each test case, output one line containing &Case #x: y&, where x is the test case number (starting from 1) and y is the sum of the maximum values in all sub-matrices of size K by K.题意：给定一个n*n矩阵，求所有k*k子矩阵中最大值之和。思路：有点类似最大子矩阵的做法。维护每列k个元素的最大值（即s[0][0]...s[k][0],s[0][1]..s[k][1]....s[0][n]...s[k][n])，然后再求这n个值每k个元素的最大值。这个问题通过deque有一个O(n)的算法。所以总体的复杂度是O(n*n)#include &cstdio&#include &string&#include &queue&#include &unordered_map&#include &algorithm&#include &iostream&#define N 3005int s[N][N],t[N];long long a[N],b[N];int T,n,c,x,k;struct node{
node(int ii,int xx):i(ii),x(xx){};
int i,x;};void mypush(deque&node& &q,node tmp){
while(!q.empty() && tmp.x&=q.back().x)
q.pop_back();
q.push_back(tmp);}long long solve(){
deque&node&
long long res = 0;
for(int i = 0;i&k-1;i++)
mypush(ans, node(i,t[i]));
for(int i = k-1;i&n;i++){
mypush(ans, node(i,t[i]));
if(ans.front().i == i-k)
ans.pop_front();
res += ans.front().x;
}}int main(){
scanf(&%d&,&T);
for(int z = 1;z&=T;z++){
long long res = 0;
deque&node& q[N];
scanf(&%d %d %d %d&,&n,&k,&c,&x);
for(int i = 0;i&n;i++)
scanf(&%lld&,&a[i]);
for(int i = 0;i&n;i++)
scanf(&%lld&,&b[i]);
for(int i = 0;i&n;i++)
for(int j = 0;j&n;j++)
s[i][j] = (a[i]*(i+1) + b[j]*(j+1) + c)%x;
for(int j = 0;j&n;j++)
for(int i = 0;i&k-1;i++)
mypush(q[j],node(i,s[i][j]));
for(int i = k-1;i&n;i++){
for(int j = 0;j&n;j++){
mypush(q[j],node(i,s[i][j]));
if(q[j].front().i == i-k)
q[j].pop_front();
t[j] = q[j].front().x;
res += solve();
printf(&Case #%d: %lld/n&,z,res);
return 0;}
最新教程周点击榜
微信扫一扫}