Act II
Research Design
| RQ | Was the problem usefully stated as a question? |
| Conceptualization | Were the details and ideas necessary to understand the problem explained, including assumptions about causal relationships? |
| Operationalization | Were the concepts and relationships carefully matched to valid empirical indicators? |
| Unit | What were the “thing” or “things” you were trying to study? Were the units directly observable, and if not, how did you plan to study them? |
| Metrics | Were the variables or features you wished to observe about your units clearly described, including the possible values they might take? |
| Instrumentation | Were the technologies, techniques, or procedures used to measure your metrics explained? |
| Method | Once measurments were taken, was it clear which statistical tests or interpretive techniques would be appropriate for analyzing the data? Would these methods allow you to answer the research question? |
| Results | Were the real or predicted outcomes of your methods presented or illustrated? |
Data Replicability
Mnemonic: qdpb or “quality discount peanut butter”
Data Quality (dq) |
Were decisions about sampling frame definitions, sample selection, and importing from sources clearly documented? |
Data Dump (dd) |
Were steps taken to prepare data for analysis explained, including formatting, cleaning, and merging queried source data? |
Data Pipeline (dp) |
Were transformations of prepared data into one or more method-specific data formats clearly explained? |
Data Base (db) |
Were the transformed, analyzed, and modeled results9 clearly presented? |
Organizational Design
| Implementation | Was it clear how the research design would be executed and by whom? |
| Organizational Context | Was the firm or organization described, including the relevant roles and relationships affecting the data scientist and decision makers? |
| Politics | Was there attention to organizational tensions or interests that might facilitate or block the implementation of the research design? |
| Biases | Were cognitive or culture assumptions or expectations that might influence the research design considered? |
| Ethics | Were questions of legality, fairness, and possible adverse or inequitable outcomes considered? Are there professional standards or formal codes of conduct that need to be addressed? |
Results are the final format of the data that most closely answer the particular research problems and from which research findings are inferred.↩