Course restructuring

Author: turukawa

Module 1 - Lesson 1 - Structuring and wrangling messy data using a spreadsheet editor.ipynb

@@ -569,7 +569,7 @@
     "    <p>The data in this folder are also sourced from the World Bank, but from a long time ago, before the World Bank knew what open or machine-readable data was. It contains some of the worst examples of data-mangling you will ever see.</p>\n",
     "</div>\n",
     "\n",
-    "Please complete this tutorial before beginning the next lesson."
+    "Please complete this tutorial before beginning the next lesson. If you are participating in a taught class, please send your tutorial submission via the required process (email or online)."
    ]
   }
  ],

Module 1 - Lesson 2 - Validating restructured data against a schema using a spreadsheet.ipynb

@@ -241,15 +241,105 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 2.2 Apply data validation to cells in a spreadsheet\n",
+    "## 2.2 Methods for data anonymisation\n",
+    "\n",
+    "There are a wide range of techniques available to support anonymisation. Broadly, though, they fit into two types:\n",
+    "\n",
+    "- __Redaction__: in which we remove fields or line-item information while maintaining sufficient integrity to permit semantic analysis;\n",
+    "- __Aggregation__: in which we deliberately aggregate data to ensure outlier anonymity;\n",
+    "\n",
+    "### 2.2.1 Redaction methods\n",
+    "\n",
+    "Before we start doing anything, we need to understand our dataset, and understand how we intend to redact it _while maintaining its internal integrity so that we can continue to conduct analysis_.\n",
+    "\n",
+    "#### Attribute suppression\n",
+    "\n",
+    "An `attribute` is also known as a `field`. This method requires that we delete an entire field. It is one of the first, and easiest, steps we can take.\n",
+    "\n",
+    "- Remove data we do not need\n",
+    "- Remove data we cannot easily redact\n",
+    "\n",
+    "This is a destructive step since suppression deletes the original data.\n",
+    "\n",
+    "#### Record suppression\n",
+    "\n",
+    "Some data are outliers; sufficiently rare that - in and of itself - these data can't be anonymised. With record suppression we remove all data related to these individuals. However, tread carefully.\n",
+    "\n",
+    "Outliers may be of significant interest if their status is part of the study. If a person's illness is unusual for the area where they live, for their ethnicity, gender or sexual orientation, then that would make them an outlier. However, that would also be important for understanding the disease.\n",
+    "\n",
+    "On the other hand, if their location, ethnicity, gender or sexual orientation have no bearing on the disease, then these could be safely redacted.\n",
+    "\n",
+    "#### Pseudonymisation\n",
+    "\n",
+    "Pseudonymisation is the replacement of identifying data with randomised values. This can be reversable, if you create a key between the data and the generated values, but irreversable if you deliberately throw aways the keys. Persistent pseudonyms support linkage between the same individual across different datasets.\n",
+    "\n",
+    "- `strings`: pseudonymise through replacement;\n",
+    "\n",
+    "#### Generalisation\n",
+    "\n",
+    "Generalisation is a deliberate reduction in the precision of data, such as converting a person's age into a range, or a precise location into a less precise location.\n",
+    "\n",
+    "- `range`: conversion of precise numbers into quantiles or statistical ranges;\n",
+    "- `cluster`: aggregation of geospatial data into statistically less significant clusters - this can also be used to mask outliers;\n",
+    "\n",
+    "Design the data ranges with appropriate sizes. Sometimes quantiles are the most appropriate, sometimes we use statistical definitions (such as geospatial ranges that are designed to include sufficient numbers of people so as to reduce deanonymisation).\n",
+    "\n",
+    "#### Shuffling\n",
+    "\n",
+    "Shuffling is where data are rearranged such that the individual attribute values are still represented in the dataset, but generally, do not correspond to the original records. This is not appropriate for all data. Swapping diseases amongst different patients will certainly render the data anonymous, but will also confuse any epidemiological analysis.\n",
+    "\n",
+    "#### Data Perturbation\n",
+    "\n",
+    "Perturbation involves adding random noise to data to \"blur\" it. This can include rounding, shifting dates, or adding geospatial displacement (jitter) to coordinate data.  This means artificially moving the precision within a small range to obscure the exact details of the person.\n",
+    "\n",
+    "- `dates`: shift exact dates by days or months;\n",
+    "- `rounding`: round off to the nearest decile or whole number, depending on the precision of the data;\n",
+    "- `coordinates`: perturb the data through geospatial displacement (jitter);\n",
+    "\n",
+    "Care must be taken not to add too little or too much perturbation.\n",
+    "\n",
+    "### 2.2.2 Aggregation methods\n",
+    "\n",
+    "Aggregation is far more destructive than is redaction. We will lose resolution on patient morphology, and we will lose the direct relationships between data in exchange for summaries of that data. But we will gain security for the individuals concerned.\n",
+    "\n",
+    "Where redaction is guided by the data almost exclusively, aggregation is guided by the research objectives for the data. Any form of aggregation will limit what can be done and awareness of these limitations is critical.\n",
+    "\n",
+    "Census data are usually aggregated in this way, with the individual microdata (responses from each household) only made available to accredited researchers, while the aggregated versions are made available to the public.\n",
+    "\n",
+    "Our objective will be to create groups of data and then perform aggregations on each group. The range of aggregations we can form include:\n",
+    "\n",
+    "- `count`: count of the individual members of the group;\n",
+    "- `totals`: sums of values, and sums of sub-groups within the values (e.g. total duration of illness, and duration of each type of illness);\n",
+    "- `averages`: including `mean`, `median` and `mode` of data sequences;\n",
+    "- `distributions`: including `quantiles`, `normals` or other types of distribution.\n",
+    "\n",
+    "The groups can be by specific `categories` or `geospatial` ranges.\n",
+    "\n",
+    "In many ways, an entire course of statistics is required to perform aggregations well.\n",
+    "\n",
+    "<div class=\"alert alert-block alert-warning\">\n",
+    "    <p><b>Aggregations require familiarity and experience with the data being aggregated.</b> It's very difficult to simply pick up a random dataset and know how to aggregate it\n",
+    "        in a way that supports analysis and extracts meaning from it. You are unlikely to be responsible for aggregating data you don't have experience with, and when you have that\n",
+    "        experience, knowing how to aggregate it will become clearer.</p>\n",
+    "</div>\n",
+    "\n",
+    "\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2.3 Apply data validation to cells in a spreadsheet\n",
     "\n",
     "Your `types` - at this stage - are only a guide. You will have no feedback, or error messages like you get when running Python code, if any of the data types in your field columns are wrong. There are a few ways to get that feedback so you can correct things, but we'll start with data validation in spreadsheet cells.\n",
     "\n",
     "The following is adapted from a [Microsoft Office tutorial](https://support.office.com/en-gb/article/apply-data-validation-to-cells-29fecbcc-d1b9-42c1-9d76-eff3ce5f7249). This approach will work in OpenOffice as well as Google Sheets, although the specific steps are different.\n",
     "\n",
     "Microsoft has an example file you can [download](http://download.microsoft.com/download/9/6/8/968A9140-2E13-4FDC-B62C-C1D98D2B0FE6/Data%20Validation%20Examples.xlsx).\n",
     "\n",
-    "### 2.2.1 Specify validation for data types\n",
+    "### 2.3.1 Specify validation for data types\n",
     "\n",
     "The process is straightforward:\n",
     "\n",
@@ -287,7 +377,7 @@
     "\n",
     "Now - only for new data - if a user tries to enter a value that is not valid, a pop-up appears with the message, \"This value doesn’t match the data validation restrictions for this cell.\" We'll run validation on your existing data shortly, but first a detour into `lists`.\n",
     "\n",
-    "### 2.2.2 Lists are a special type\n",
+    "### 2.3.2 Lists are a special type\n",
     "\n",
     "Before you can validate a `list` type, you need to specify valid terms. In Excel, this requires an [extra set of steps](https://support.office.com/en-us/article/create-a-drop-down-list-7693307a-59ef-400a-b769-c5402dce407b).\n",
     "\n",
@@ -306,7 +396,7 @@
     "  - Convert your list to a table with __Ctrl+T__, then from the __Table Design__ tab give your table a name, permitting you to reference the table name and column (e.g. `=CityTable[City]`)\n",
     "  - From the __Formulas__ tab select __Name Manager__, create a __New__ item with an appropriate name (e.g. `CityList`), and reference the cells (e.g. `=Sheet1!A4:A10`), which then lets you reference your list anywhere (e.g. `=CityList`)\n",
     "\n",
-    "### 2.2.3 Validate and get error messages for your existing data\n",
+    "### 2.3.3 Validate and get error messages for your existing data\n",
     "\n",
     "After you've specified validation rules on your existing data you might be disappoined. Excel does not automatically notify you whether these cells contain invalid data. Here's a quick way to [highlight existing invalid cells](https://support.office.com/en-us/article/more-on-data-validation-f38dee73-9900-4ca6-9301-8a5f6e1f0c4c) by circling the values:\n",
     "\n",
@@ -348,7 +438,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 2.3 Saving your validated file as a comma-separated-value\n",
+    "## 2.4 Saving your validated file as a comma-separated-value\n",
     "\n",
     "Comma separated value files (`.csv`) are text files in which the comma character `,` separates each field of text. Where a comma appears in the value - whether a `string` or `number` - the value is then surrounded by quotation marks, e.g. `100, 200, \"20,000\"` indicates three values in three separate fields.\n",
     "\n",
@@ -372,7 +462,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 2.4 Validating your data and JSON schema using CSVLint\n",
+    "## 2.5 Validating your data and JSON schema using CSVLint [optional]\n",
     "\n",
     "In the next lesson, we'll learn how to validate your data using Python directly in a Jupyter Notebook, for now we'll use an online resource provided by the Open Data Institute called [CSVLint](https://csvlint.io/).\n",
     "\n",
@@ -416,7 +506,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 2.5 Lesson tutorial\n",
+    "## 2.6 Lesson tutorial\n",
     "\n",
     "<div class=\"alert alert-block alert-success\">\n",
     "    <p><b>Tutorial:</b></p>\n",
@@ -429,7 +519,7 @@
     "    </ul>\n",
     "</div>\n",
     "\n",
-    "Please complete the tutorial before continuing with this series."
+    "Please complete the tutorial before continuing with this series. If you are participating in a taught class, please send your tutorial submission via the required process (email or online)."
    ]
   }
  ],

Module 1 - Leçon 1 - Structurer et organiser des données désordonnées à l'aide d'un tableur.ipynb

@@ -554,7 +554,7 @@
     "    <p>Les données de ce dossier proviennent également de la Banque mondiale, mais elles datent d'une époque bien antérieure, avant que la Banque mondiale ne sache ce qu'étaient des données ouvertes ou exploitables par une machine. Il contient certains des pires exemples de manipulation de données que vous n'aurez jamais vus.</p>\n",
     "</div>\n",
     "\n",
-    "Veuillez compléter ce tutoriel avant de commencer la prochaine leçon."
+    "Veuillez compléter ce tutoriel avant de commencer la prochaine leçon. Si vous participez à un cours, veuillez envoyer votre demande de tutorat par le biais du processus requis (par courriel ou en ligne)."
    ]
   }
  ],