Last modified: 2014-10-31 12:52:51 UTC
The three bits partitions [1] on 2014-10-07T1[789]:xx:xx, were not marked successful. What happened? [1] _________________________________________________________________ qchris@stat1002 // jobs: 0 // time: 13:05:11 // exit code: 0 cwd: ~/cluster-scripts ./dump_webrequest_status.sh +---------------------+--------+--------+--------+--------+ | Date | bits | text | mobile | upload | +---------------------+--------+--------+--------+--------+ [...] | 2014-10-07T15:xx:xx | . | . | . | . | | 2014-10-07T16:xx:xx | . | . | . | . | | 2014-10-07T17:xx:xx | X | . | . | . | | 2014-10-07T18:xx:xx | X | . | . | . | | 2014-10-07T19:xx:xx | X | . | . | . | | 2014-10-07T20:xx:xx | . | . | . | . | | 2014-10-07T21:xx:xx | . | . | . | . | [...] +---------------------+--------+--------+--------+--------+ Statuses: . --> Partition is ok X --> Partition is not ok (duplicates, missing, or nulls)
Since checking missings across hour boundaries does not make the missing go away [1], it does not seem to be the neighbored race condition described in bug 69615. Since it seems only esams bits are affected, it might be another instance of bug 71435. More investigation needed. [1] _________________________________________________________________ qchris@stat1002 // jobs: 0 // time: 13:17:03 // exit code: 0 cwd: ~/refinery hive -f three_hour_stats.hql -d table=wmf_raw.webrequest -d webrequest_source=bits -d year=2014 -d month=10 -d day=7 -d hourA=17 -d hourB=18 -d hourC=19 [...] Total MapReduce CPU Time Spent: 0 days 4 hours 9 minutes 5 seconds 150 msec OK hostname sequence_min sequence_max count_actual count_expected count_different count_duplicate count_null_sequence percent_different cp3022.esams.wikimedia.org 2082145621 2135356589 52052666 53210969 1158303 0 0 -2.1768124538382305 cp3021.esams.wikimedia.org 2078380896 2131585483 52714377 53204588 490211 0 0 -0.9213697886355214 cp3020.esams.wikimedia.org 1828915695 1882208121 50611125 53292427 2681302 0 0 -5.031300225827583 cp3019.esams.wikimedia.org 1830855032 1884131067 51714434 53276036 1561602 0 0 -2.931152760689628 Time taken: 473.312 seconds, Fetched: 4 row(s)