March 21, 2016By Fidel Beraldi, Fraud and Risk Department, First Data Brazil and Alair Pereira do Lago, Department of Computer Science, Universidade de São Paulo Editor’s note: Fraud prevention technology continues to become more sophisticated as e-commerce merchants and those trying to steal from them move and countermove. But, the research behind advances in fraud prevention is not always visible, even within the industry. Fidel Beraldi, a fraud and risk manager with First Data in Brazil has agreed to share recent research conducted by Dr. Alair Pereira do Lago, a faculty member in the Department of Computer Science at the Universidade de São Paulo on a new statistical model that can be applied to fraud prevention that will enable fraud scoring systems to adapt to shifting fraud techniques quicker. While the tone of this article is academic, it has been adapted from a much longer research report. It does retain some statistical language that is more sophisticated than usually appears on CardNotPresent.com. Readers interested in clarification or discussing the results in more depth with the authors can contact Fidel Beraldi at firstname.lastname@example.org . Fraud indicators have shown that card-not-present transactions are riskier than card-present transactions, due to the fact that neither the cardholder nor the card itself is physically present at the point of sale. In these scenarios, more opportunities are created for fraudsters to produce new methods, resulting in large losses for the financial system. As fraudsters quickly adapt to existing fraud prevention measures, statistical models for fraud detection also need to be adaptable and flexible to change over time in a dynamic way. Fraud scoring models can be updated sporadically or continuously over time, which raises the question of dynamic update of the model parameters to detect fraud.
[hide for=”!logged”]Adrian E. Raftery, a professor of statistics and sociology at the University of Washington has developed a new method called Dynamic Model Averaging (DMA) that implements a process of continuous updating over time. The DMA methodology combines some existing ideas: weighted Bayesian models (Bayesian Model Averaging – BMA), Markov chains and forgetting factor in shaping state-space. However, this model had never been applied on a credit card fraud dataset before. This article shows results regarding the application of DMA methodology to a credit card fraud dataset in Brazil. Experiment For the experiment, an e-commerce company provided a fraud dataset that had been generated by its payment system. The data analysis was performed following a non-disclosure agreement, as recommended by the PCI Data Security Standard. Data analysis made it possible to compare the DMA methodology against the classical logistic regression model, which is often used in fraud detection process in many risk systems. The database for the experiment is composed of credit card transactions posted between July 2009 and January 2014. For each extracted record, there were 52 independent variables and one dependent variable (fraud/non-fraud transaction). Denied transactions, regardless of the reason, were excluded from the database. Thus, only approved transactions were considered during the modeling process. Fraudulent transactions were identified in two ways: through chargebacks and by the analysis of the company’s internal team. However, both ways of detection resulted in a classification of fraud transactions in the final database. Transactions not classified as fraud were identified as non-fraud. For the modeling process, we sampled 428,256 records, with 22,615 fraud and 405,641 non-fraud transactions. This implies a ratio of about 1 fraud transaction for each 18 non-fraud transactions. This ratio of fraud and non-fraud is very close to the values adopted for this kind of problem, as noted in previous experiments. After collecting the data for the modeling process, the sampled records were split into 80 percent (342,605) for model development and 20 percent (85,651) for validation. Results The performance of the logistic regression model and the DMA model are separately evaluated on the table below. In general, it is noticed that the DMA model presents better performance for all indicators when compared to logistic regression models, except for the detection rate, which shows a difference of 6.4 percent in favor of the latter. Overall, it is observed that the DMA model shows better classification performance than the logistic regression model.
|Model Performance Indicators||Models|
|Logistic Regression||DMA Model|
|Detection Rate – Transactions classified as fraudulent by the model divided by total fraudulent transactions||62.9%||56.5%|
|False Positive Rate – Non fraudulent transactions classified as fraudulent transactions by model divided by total non-fraudulent transactions||16.9%||2.1%|
|KS measure – Classification performance||48.1%||70.4%|
|Accuracy – Proportion of transactions classified correctly||82.1%||95.7%|
|F-measure – Weighted average of precision and recall||27.0%||58.2%|