Others

Four Great Tips for Data Mining

In this guide
  1. 1. Collect as much data as possible from multiple sources
  2. 2. Use a clear sampling strategy
  3. 3. Make sure to use ‘throwaway modeling’
  4. 4. Refresh your models regularly
  5. Conclusion

Data mining has become a critical analytical tool for managers, marketers, digital advertisers and anyone interested in exploring hidden but valuable patterns in large data sets. It is much like mining diamonds from the earth.

It is used to extract knowledge and actionable information and to extrapolate patterns from the data collected. It is employed in a vast range of fields: from analysing a population’s healthcare data to assess its susceptibility to a pandemic to carrying out market basket analysis, from fraud detection to web scraping, and from corporate budgeting to detecting the political and social mood of the population.

By applying the latest analytics tools, we can get data-backed insights into phenomena that were once simply ‘hunches,’ and work towards improving or fixing current processes or building a new system.

With data mining, a digital marketer can find groups of people with similar preferences and target them more effectively and accurately. Similarly, an online gaming company can identify new trends and demands among its potential users on social media and make the necessary updates to its games.

As with any tool that gives a business user deeper insights, you may be tempted to overuse data mining. But it must be used judiciously, and not to twist facts to suit your needs. To maximize your chances of success, focus on projects aligned with measurable and identifiable business goals, like improving customer engagement and loyalty, identifying cross-selling opportunities or detecting digital advertising fraud.

The following are the four most important tips for knowledge discovery:

1. Collect as much data as possible from multiple sources

Data from a single source or only internal sources is valuable and essential, but it will not reveal much about, say, a customer. If you limit yourself to your internal data, you may miss out on a vast amount of user activity beyond the limits of your system. If you can trace your customers’ digital footprints (their social media activity, browser searches, reviews and so on), analytics can reveal actionable information that leads to better engagement and increased revenue. You can model customer behavior and profiles and find innovative ways to influence them, such as identifying and using a user’s influential associates. With the right mix of communication, you can build a favorable public perception in each target group. Creating a training set from the wrong sources means you may skew your model and make it unreliable.

2. Use a clear sampling strategy

Sampling is the art of choosing the right, limited number of items from a broad universe so that the sample is most likely to represent it. Many powerful analytics tools and applications have failed because they were not given a proper sample of data to work with: it was too small, too limited in attributes or too biased. A clear, concise and unambiguous sampling strategy is the key to successful data mining. For example, to uncover digital advertising fraud, you would scrape data from known fake websites with fake visitors but real ad revenue. To find the general mood of young people on climate change, you would mine data from popular social media platforms. Always make sure to keep a “holdout sample” and do not use it to train the model. It is a reference sample used to test the model’s predictive performance, analysis and efficacy after it has been trained. Testing ensures that the model does not make wrong predictions and has no bias.

3. Make sure to use ‘throwaway modeling’

The first step in designing a model is identifying the independent variables, or predictors, from the overall set available. It is always best to use throwaway modeling, or rapid modeling, to create a model for the client’s initial presentation after gathering preliminary requirements. It helps you visualize the system and come up with more detailed requirements and expected outcomes. Throwaway modeling requires you to “throw” in all the data, test all alternative models and find the best fit using a systematic selection process. It helps you reduce the number of ‘reset’ and ‘redesign’ decisions at a later stage, when they are more complex and time-consuming. If you skip this step, you may leave out some important relationships and patterns in your data. That can be detrimental to the overall success and expected outcomes of the project. A well-structured and intelligently built throwaway model will improve overall productivity.

4. Refresh your models regularly

The real world is dynamic, and the data that represents it is only accurate for a short point in time. To remain relevant, your predictive model must regularly refresh its data and remodel itself to accommodate changing realities. Even the best predictive models do not fit real-world data forever. Therefore, the refresh rate for models (the frequency at which data is updated and added and the model is recalibrated) needs to be in sync with the modeling system. It can be updated weekly, daily or based on certain triggers linked to important events. Depending on the system, you may need to factor in only the new data, or the calculations may have to be done all over again every time. The correct frequency for these updates is critical not only for the model’s validity but also for its efficiency.

Conclusion

In the end, it is all about people. Make sure that the insights you present are meaningful to them. When communicating your conclusions and predictions to the organization, you must understand which details and formats to use in the presentation (pictures, graphs, patterns, word clouds and so on) so that non-technical users can understand them. Excessive jargon and mathematical equations may be too complicated for others to understand and use in decision and policy making. Your model must have real applications in the real world. A theoretical model with no practical use may not have any takers at all. Set out clear execution paths for all intended users, because something that cannot be demonstrated may as well not exist.

Written by

David Smith

Published on TheTechRim in Others.