Outliers

Outlier is an observation that lies out side the overall pattern of distribution . It will be numerically distant from rest of the data. Outlier is "one that appears to deviate markedly from other members of the sample in which it occurs." says Grubbs.Usually, the presence of an outlier indicates some sort of problem.Outlier points therefore indicates faulty data, erroneous procedures, or areas where a certain theory might be invalid.It can be due to some experimental errors or by measurement errors, or sometimes by long-tailed population.

If it is measurement error ,identification and removal of that outlying data is widely preferred. Because the presence of outliers will impact on the accuracy and appropriateness of a result.  Quartile method is used to identify outliers.

If it is long-tailed population ,it means that the distribution has a high kurtosis (kurtosis means the peak level of probability distribution of any real-valued random variable). So, while dealing with it is necessary for one to be careful and cautious in using tools or intuitions that assume a normal distribution

Causes Of Outliers:

The presence of outliers is due to one of the following reasons :

• Aberrations in system behaviour ,instrument error,or human error,
• Error in data transmission or transcription. 
• A transient malfunction in the physical apparatus
• As a result of flaw in the assumed theory
• Natural deviation in the population.

Outlier Detection:

Basically there are three to the problem of outlier detection:

• Determination of outliers without a prior knowledge of data : In this approach , data is processed as a static distribution. And remote points will be pinpointed and flagged as potential outliers.
• Model both normality and abnormality : This approach requires a pre-labeled data, tagged as normal or abnormal.
• Model only normality (or in a few cases model abnormality)  :One might consider it as semi-supervised as the normal class is taught .But the algorithm learns to recognize abnormality.

Apart from this , Grubb used a separate method called the ESD method for the detection of outliers where to detect a outlier from a sample of data, one must first find how far the outlier is from the rest of the data? The difference between the outlier and the mean divided by the SD is found . If that value  is large, then the outlier is far from the others.

In a large sample of data , small number of outliers are to be expected. Such outliers can easily be found in histograms .They may include the sample maximum or sample minimum, or at times both, based on whether they are extremely high or extremely low. There wil be cases where the sample maximum & sample minimum will not be outliers when are unusually far from other observations