fastNLP/tutorials/fastnlp_tutorial_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "213d538c",
   "metadata": {},
   "source": [
    "# T3. dataloader 的内部结构和基本使用\n",
    "\n",
    "  1   fastNLP 中的 dataloader\n",
    " \n",
    "    1.1   dataloader 的基本介绍\n",
    "\n",
    "    1.2   dataloader 的函数创建\n",
    "\n",
    "  2   fastNLP 中 dataloader 的延伸\n",
    "\n",
    "    2.1   collator 的概念与使用\n",
    "\n",
    "    2.2   sampler 的概念与使用"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85857115",
   "metadata": {},
   "source": [
    "## 1. fastNLP 中的 dataloader\n",
    "\n",
    "### 1.1 dataloader 的基本介绍\n",
    "\n",
    "在`fastNLP 0.8`的开发中，最关键的开发目标就是**实现`fastNLP`对当前主流机器学习框架**，例如\n",
    "\n",
    "  **较为火热的`pytorch`**，以及**国产的`paddle`和`jittor`的兼容**，扩大受众的同时，也是助力国产\n",
    "\n",
    "本着分而治之的思想，我们可以将`fastNLP 0.8`对`pytorch`、`paddle`、`jittor`框架的兼容，划分为\n",
    "\n",
    "    **对数据预处理**、**批量`batch`的划分与补齐**、**模型训练**、**模型评测**，**四个部分的兼容**\n",
    "\n",
    "  针对数据预处理，我们已经在`tutorial-1`中介绍了`dataset`和`vocabulary`的使用\n",
    "\n",
    "    而结合`tutorial-0`，我们可以发现**数据预处理环节本质上是框架无关的**\n",
    "\n",
    "    因为在不同框架下，读取的原始数据格式都差异不大，彼此也很容易转换\n",
    "\n",
    "只有涉及到张量、模型，不同框架才展现出其各自的特色：**`pytorch`中的`tensor`和`nn.Module`**\n",
    "\n",
    "    **在`paddle`中称为`tensor`和`nn.Layer`**，**在`jittor`中则称为`Var`和`Module`**\n",
    "\n",
    "    因此，**模型训练、模型评测**，**是兼容的重难点**，我们将会在`tutorial-5`中详细介绍\n",
    "\n",
    "  针对批量`batch`的处理，作为`fastNLP 0.8`中框架无关部分想框架相关部分的过渡\n",
    "\n",
    "    就是`dataloader`模块的职责，这也是本篇教程`tutorial-3`讲解的重点\n",
    "\n",
    "**`dataloader`模块的职责**，详细划分可以包含以下三部分，**采样划分、补零对齐、框架匹配**\n",
    "\n",
    "    第一，确定`batch`大小，确定采样方式，划分后通过迭代器即可得到`batch`序列\n",
    "\n",
    "    第二，对于序列处理，这也是`fastNLP`主要针对的，将同个`batch`内的数据对齐\n",
    "\n",
    "    第三，**`batch`内数据格式要匹配框架**，**但`batch`结构需保持一致**，**参数匹配机制**\n",
    "\n",
    "  对此，`fastNLP 0.8`给出了 **`TorchDataLoader`、`PaddleDataLoader`和`JittorDataLoader`**\n",
    "\n",
    "    分别针对并匹配不同框架，但彼此之间参数名、属性、方法仍然类似，前两者大致如下表所示\n",
    "\n",
    "| <div align=\"center\">名称</div> | <div align=\"center\">参数</div> | <div align=\"center\">属性</div> | <div align=\"center\">功能</div> | <div align=\"center\">内容</div> |\n",
    "|:--|:--:|:--:|:--|:--|\n",
    "| **`dataset`** | √ | √ | 指定`dataloader`的数据内容  |  |\n",
    "| `batch_size` | √ | √ | 指定`dataloader`的`batch`大小 | 默认`16` |\n",
    "| `shuffle` | √ | √ | 指定`dataloader`的数据是否打乱 | 默认`False` |\n",
    "| `collate_fn` | √ | √ | 指定`dataloader`的`batch`打包方法 | 视框架而定 |\n",
    "| `sampler` | √ | √ | ？ | 默认`None` |\n",
    "| `batch_sampler` | √ | √ | ？ | 默认`None` |\n",
    "| `drop_last` | √ | √ | 指定`dataloader`划分`batch`时是否丢弃剩余的 | 默认`False` |\n",
    "| `cur_batch_indices` |  | √ | 记录`dataloader`当前遍历批量序号 |  |\n",
    "| `num_workers` | √ | √ | 指定`dataloader`开启子进程数量 | 默认`0` |\n",
    "| `worker_init_fn` | √ | √ | 指定`dataloader`子进程初始方法 | 默认`None` |\n",
    "| `generator` | √ | √ | 指定`dataloader`子进程随机种子 | 默认`None` |\n",
    "| `prefetch_factor` |  | √ | 指定为每个`worker`装载的`sampler`数量 | 默认`2` |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60a8a224",
   "metadata": {},
   "source": [
    "&emsp; 论及`dataloader`的函数，其中，`get_batch_indices`用来获取当前遍历到的`batch`序号，其他函数\n",
    "\n",
    "&emsp; &emsp; 包括`set_ignore`、`set_pad`和`databundle`类似，请参考`tutorial-2`，此处不做更多介绍\n",
    "\n",
    "&emsp; &emsp; 以下是`tutorial-2`中已经介绍过的数据预处理流程，接下来是对相关数据进行`dataloader`处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "aca72b49",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Processing:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Processing:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Processing:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+------------------+-----------+------------------+--------------------+--------------------+\n",
      "| SentenceId | Sentence         | Sentiment | input_ids        | token_type_ids     | attention_mask     |\n",
      "+------------+------------------+-----------+------------------+--------------------+--------------------+\n",
      "| 5          | A comedy-dram... | positive  | [101, 1037, 4... | [0, 0, 0, 0, 0,... | [1, 1, 1, 1, 1,... |\n",
      "| 2          | This quiet , ... | positive  | [101, 2023, 4... | [0, 0, 0, 0, 0,... | [1, 1, 1, 1, 1,... |\n",
      "| 1          | A series of e... | negative  | [101, 1037, 2... | [0, 0, 0, 0, 0,... | [1, 1, 1, 1, 1,... |\n",
      "| 6          | The Importanc... | neutral   | [101, 1996, 5... | [0, 0, 0, 0, 0,... | [1, 1, 1, 1, 1,... |\n",
      "+------------+------------------+-----------+------------------+--------------------+--------------------+\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append('..')\n",
    "\n",
    "import pandas as pd\n",
    "from functools import partial\n",
    "from fastNLP.transformers.torch import BertTokenizer\n",
    "\n",
    "from fastNLP import DataSet\n",
    "from fastNLP import Vocabulary\n",
    "from fastNLP.io import DataBundle\n",
    "\n",
    "\n",
    "class PipeDemo:\n",
    "    def __init__(self, tokenizer='bert-base-uncased'):\n",
    "        self.tokenizer = BertTokenizer.from_pretrained(tokenizer)\n",
    "\n",
    "    def process_from_file(self, path='./data/test4dataset.tsv'):\n",
    "        datasets = DataSet.from_pandas(pd.read_csv(path, sep='\\t'))\n",
    "        train_ds, test_ds = datasets.split(ratio=0.7)\n",
    "        train_ds, dev_ds = datasets.split(ratio=0.8)\n",
    "        data_bundle = DataBundle(datasets={'train': train_ds, 'dev': dev_ds, 'test': test_ds})\n",
    "\n",
    "        encode = partial(self.tokenizer.encode_plus, max_length=100, truncation=True,\n",
    "                         return_attention_mask=True)\n",
    "        data_bundle.apply_field_more(encode, field_name='Sentence', progress_bar='tqdm')\n",
    "        \n",
    "        target_vocab = Vocabulary(padding=None, unknown=None)\n",
    "\n",
    "        target_vocab.from_dataset(*[ds for _, ds in data_bundle.iter_datasets()], field_name='Sentiment')\n",
    "        target_vocab.index_dataset(*[ds for _, ds in data_bundle.iter_datasets()], field_name='Sentiment',\n",
    "                                   new_field_name='target')\n",
    "\n",
    "        data_bundle.set_pad('input_ids', pad_val=self.tokenizer.pad_token_id)\n",
    "        data_bundle.set_ignore('SentenceId', 'Sentence', 'Sentiment')  \n",
    "        return data_bundle\n",
    "\n",
    "    \n",
    "pipe = PipeDemo(tokenizer='bert-base-uncased')\n",
    "\n",
    "data_bundle = pipe.process_from_file('./data/test4dataset.tsv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76e6b8ab",
   "metadata": {},
   "source": [
    "### 1.2 dataloader 的函数创建\n",
    "\n",
    "在`fastNLP 0.8`中，**更方便、可能更常用的`dataloader`创建方法是通过`prepare_xx_dataloader`函数**\n",
    "\n",
    "&emsp; 例如下方的`prepare_torch_dataloader`函数，指定必要参数，读取数据集，生成对应`dataloader`\n",
    "\n",
    "&emsp; 类型为`TorchDataLoader`，只能适用于`pytorch`框架，因此对应`trainer`初始化时`driver='torch'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5fd60e42",
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastNLP import prepare_torch_dataloader\n",
    "\n",
    "train_dataset = data_bundle.get_dataset('train')\n",
    "evaluate_dataset = data_bundle.get_dataset('dev')\n",
    "\n",
    "train_dataloader = prepare_torch_dataloader(train_dataset, batch_size=16, shuffle=True)\n",
    "evaluate_dataloader = prepare_torch_dataloader(evaluate_dataset, batch_size=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c53f181",
   "metadata": {},
   "source": [
    "```python\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    train_dataloader=train_dataloader,\n",
    "    optimizers=optimizer,\n",
    "\t...\n",
    "\tdriver='torch',\n",
    "\tdevice='cuda',\n",
    "\t...\n",
    "    evaluate_dataloaders=evaluate_dataloader,     \n",
    "    metrics={'acc': Accuracy()},\n",
    "\t...\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f457a6e",
   "metadata": {},
   "source": [
    "之所以称`prepare_xx_dataloader`函数更方便，是因为其**导入对象不仅可也是`DataSet`类型**，**还可以**\n",
    "\n",
    "&emsp; **是`DataBundle`类型**，不过数据集名称需要是`'train'`、`'dev'`、`'test'`供`fastNLP`识别\n",
    "\n",
    "&emsp; 例如下方就是**直接通过`prepare_paddle_dataloader`函数生成基于`PaddleDataLoader`的字典**\n",
    "\n",
    "&emsp; 在接下来`trainer`的初始化过程中，按如下方式使用即可，除了初始化时`driver='paddle'`外\n",
    "\n",
    "&emsp; &emsp; 这里也可以看出 **`evaluate_dataloaders`的妙处**，一次评测可以针对多个数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7827557d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastNLP import prepare_paddle_dataloader\n",
    "\n",
    "dl_bundle = prepare_paddle_dataloader(data_bundle, batch_size=16, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d898cf40",
   "metadata": {},
   "source": [
    "```python\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    train_dataloader=dl_bundle['train'],\n",
    "    optimizers=optimizer,\n",
    "\t...\n",
    "\tdriver='paddle',\n",
    "\tdevice='gpu',\n",
    "\t...\n",
    "    evaluate_dataloaders={'dev': dl_bundle['dev'], 'test': dl_bundle['test']},     \n",
    "    metrics={'acc': Accuracy()},\n",
    "\t...\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d74d0523",
   "metadata": {},
   "source": [
    "## 2. fastNLP 中 dataloader 的延伸\n",
    "\n",
    "### 2.1 collator 的概念与使用\n",
    "\n",
    "在`fastNLP 0.8`中，在数据加载模块`DataLoader`之前，还存在其他的一些模块，负责例如对文本数据\n",
    "\n",
    "&emsp; 进行补零对齐，即 **核对器`collator`模块**，进行分词标注，即 **分词器`tokenizer`模块**\n",
    "\n",
    "&emsp; 本节将对`fastNLP`中的核对器`collator`等展开介绍，分词器`tokenizer`将在下一节中详细介绍\n",
    "\n",
    "在`fastNLP 0.8`中，**核对器`collator`模块负责文本序列的补零对齐**，通过"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "651baef6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from fastNLP import prepare_torch_dataloader\n",
    "\n",
    "dl_bundle = prepare_torch_dataloader(data_bundle, train_batch_size=2)\n",
    "\n",
    "print(type(dl_bundle), type(dl_bundle['train']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f816ef5",
   "metadata": {},
   "source": [
    "&emsp; "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "726ba357",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dataloader = prepare_torch_dataloader(datasets['train'], train_batch_size=2)\n",
    "print(type(dataloader))\n",
    "print(dir(dataloader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0795b3e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dataloader.collate_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9bbd9a7",
   "metadata": {},
   "source": [
    "### 2.2 sampler 的概念与使用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0c3c58d",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dataloader.batch_sampler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51bf0878",
   "metadata": {},
   "source": [
    "&emsp; "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fd2486f",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}