Skip to content

Commit 75642e9

Browse files
authored
Support parsing CSV by WASM (#70)
* Prepare Rust support for parsing CSV * Update Rust setup in CI/CD workflows * Update Rust setup action * Update CI configuration to target wasm32-unknown-unknown * Fix CI * Add wasm-opt dependency * Add libstdc++-12-dev installation to CI workflow * Add libstdc++-12-dev installation step * Update libstdc++-12-dev installation in CI/CD workflows * Update CI configuration and dependencies * Add artifact upload step and update import path * Add dependency on build job in test_deno workflow * Add WASM CSV parsing functionality * Remove rust-cache action * Change build method * Improve building WASM and loading WASM from Node.js * Add wasm-pack dependency * Add wasm-pack installation to CI/CD workflows * Add wasm-pack installation step to CI workflow * Refactor vite-plugin-wasm-pack configuration * Update dependabot configuration * Add Rust checking workflow * Add clippy component to Rust CI workflow * Add linting and formatting on git stageed * Update cargo commands in ci.yaml * Add rustfmt component to CI workflow * Update parseStringToArraySyncWASM function and add fast-check library for property-based testing * Add doc for experimental WebAssembly parsing support for high performance * Update package.json dependencies * Create yellow-chefs-run.md * Update changeset
1 parent ce5f78c commit 75642e9

23 files changed

+2862
-269
lines changed

.changeset/yellow-chefs-run.md

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
"web-csv-toolbox": minor
3+
---
4+
5+
Support parsing CSV by WASM build by Rust
6+
7+
- **New Features**
8+
- Introduced WebAssembly support for high-performance CSV parsing in the CSV Toolbox, including new APIs and limitations.
9+
- Added a weekly update schedule for cargo package dependencies.
10+
- Implemented a Vite plugin for integrating WebAssembly modules into projects.
11+
12+
- **Enhancements**
13+
- Added new configuration rule for Rust files, setting indent size to 4 spaces.
14+
- Enhanced continuous integration and deployment workflows with additional steps for Rust and WebAssembly setup.
15+
- Updated documentation to reflect WebAssembly features and usage in CSV parsing.
16+
17+
- **Chores**
18+
- Introduced linting and formatting checks for JavaScript, TypeScript, JSON, and Rust files.

.editorconfig

+3
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,6 @@ end_of_line = lf
1010
charset = utf-8
1111
trim_trailing_whitespace = true
1212
insert_final_newline = true
13+
14+
[*.rs]
15+
indent_size = 4

.github/dependabot.yml

+6-2
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,11 @@
55

66
version: 2
77
updates:
8-
- package-ecosystem: npm # See documentation for possible values
9-
directory: "/" # Location of package manifests
8+
- package-ecosystem: npm
9+
directory: "/"
10+
schedule:
11+
interval: "weekly"
12+
- package-ecosystem: cargo
13+
directory: "/web-csv-toolbox-wasm"
1014
schedule:
1115
interval: "weekly"

.github/workflows/cd.yaml

+10
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,16 @@ jobs:
2525
- uses: pnpm/action-setup@v2
2626
with:
2727
version: 8
28+
- run: |
29+
sudo apt-get update
30+
sudo apt-get install libstdc++-12-dev
31+
- name: Install latest
32+
uses: moonrepo/setup-rust@v1
33+
with:
34+
targets: wasm32-unknown-unknown
35+
channel: nightly
36+
- name: Install wasm-pack
37+
run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
2838
- name: Setup Node.js 20
2939
uses: actions/setup-node@v3
3040
with:

.github/workflows/ci.yaml

+76-2
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,16 @@ jobs:
1111
steps:
1212
- name: Checkout Repo
1313
uses: actions/checkout@v4
14+
- run: |
15+
sudo apt-get update
16+
sudo apt-get install libstdc++-12-dev
17+
- name: Install latest
18+
uses: moonrepo/setup-rust@v1
19+
with:
20+
targets: wasm32-unknown-unknown
21+
channel: nightly
22+
- name: Install wasm-pack
23+
run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
1424
- uses: pnpm/action-setup@v2
1525
with:
1626
version: 8
@@ -23,6 +33,10 @@ jobs:
2333
run: pnpm install --frozen-lockfile
2434
- name: Build
2535
run: pnpm run build
36+
- uses: actions/upload-artifact@v4
37+
with:
38+
name: dist
39+
path: dist
2640
test_nodejs:
2741
runs-on: ubuntu-latest
2842
strategy:
@@ -40,20 +54,29 @@ jobs:
4054
with:
4155
node-version: ${{ matrix.node-version }}
4256
cache: pnpm
57+
- name: Install latest
58+
uses: moonrepo/setup-rust@v1
59+
with:
60+
targets: wasm32-unknown-unknown
61+
channel: nightly
62+
- name: Install wasm-pack
63+
run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
4364
- run: pnpm install --frozen-lockfile
4465
- run: pnpm run build
4566
- run: pnpm test run
4667

4768
test_deno:
4869
runs-on: ubuntu-latest
70+
needs: build
4971
steps:
5072
- uses: actions/checkout@v3
73+
- uses: actions/download-artifact@v4
5174
- uses: denoland/setup-deno@v1
5275
with:
5376
deno-version: v1.x
5477
- run: |
5578
deno eval '
56-
import { parse } from "./src/web-csv-toolbox.ts";
79+
import { parse } from "./dist/es/web-csv-toolbox.js";
5780
5881
const csv = `name,age
5982
Alice,42
@@ -73,6 +96,16 @@ jobs:
7396
- firefox
7497
steps:
7598
- uses: actions/checkout@v3
99+
- run: |
100+
sudo apt-get update
101+
sudo apt-get install libstdc++-12-dev
102+
- name: Install latest
103+
uses: moonrepo/setup-rust@v1
104+
with:
105+
targets: wasm32-unknown-unknown
106+
channel: nightly
107+
- name: Install wasm-pack
108+
run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
76109
- uses: pnpm/action-setup@v2
77110
with:
78111
version: 8
@@ -97,6 +130,13 @@ jobs:
97130
# - safari
98131
steps:
99132
- uses: actions/checkout@v3
133+
- name: Install latest
134+
uses: moonrepo/setup-rust@v1
135+
with:
136+
targets: wasm32-unknown-unknown
137+
channel: nightly
138+
- name: Install wasm-pack
139+
run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
100140
- uses: pnpm/action-setup@v2
101141
with:
102142
version: 8
@@ -121,6 +161,13 @@ jobs:
121161
- edge
122162
steps:
123163
- uses: actions/checkout@v3
164+
- name: Install latest
165+
uses: moonrepo/setup-rust@v1
166+
with:
167+
targets: wasm32-unknown-unknown
168+
channel: nightly
169+
- name: Install wasm-pack
170+
run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
124171
- uses: pnpm/action-setup@v2
125172
with:
126173
version: 8
@@ -129,10 +176,12 @@ jobs:
129176
with:
130177
node-version: 20
131178
cache: pnpm
179+
- name: Install wasm-pack
180+
run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
132181
- run: pnpm install --frozen-lockfile
133182
- run: pnpm test run -- --browser.name=${{ matrix.browsers }} --browser.headless
134183

135-
check:
184+
check_nodejs:
136185
name: Check
137186
runs-on: ubuntu-latest
138187
permissions:
@@ -152,3 +201,28 @@ jobs:
152201
run: pnpm install --frozen-lockfile
153202
- name: Check
154203
run: npx biome ci .
204+
check_rust:
205+
name: Check Rust
206+
runs-on: ubuntu-latest
207+
permissions:
208+
contents: read
209+
steps:
210+
- name: Checkout Repo
211+
uses: actions/checkout@v4
212+
- run: |
213+
sudo apt-get update
214+
sudo apt-get install libstdc++-12-dev
215+
- name: Install latest
216+
uses: moonrepo/setup-rust@v1
217+
with:
218+
targets: wasm32-unknown-unknown
219+
channel: nightly
220+
components: clippy,rustfmt
221+
- name: Install wasm-pack
222+
run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
223+
- name: Run Clippy
224+
run: cargo clippy --manifest-path=./web-csv-toolbox-wasm/Cargo.toml --all-targets --all-features
225+
env:
226+
RUSTFLAGS: -D warnings
227+
- name: Run Fmt
228+
run: cargo fmt --manifest-path=./web-csv-toolbox-wasm/Cargo.toml --all -- --check

.lintstagedrc.js

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
export default {
2+
"*.{js,ts,json}": "biome check --no-errors-on-unmatched --apply",
3+
"*.rs": [
4+
() => "cargo fmt --manifest-path=./web-csv-toolbox-wasm/Cargo.toml --all",
5+
() => "cargo clippy --manifest-path=./web-csv-toolbox-wasm/Cargo.toml --all-targets --all-features",
6+
],
7+
};

.lintstagedrc.yaml

-2
This file was deleted.

README.md

+39
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,10 @@ A CSV Toolbox utilizing Web Standard APIs.
5555
- 🔄 Flexible BOM handling.
5656
- 🗜️ Supports various compression formats.
5757
- 🔤 Charset specification for diverse encoding.
58+
- 📦 **Lightweight and Zero Dependencies**: No external dependencies, only Web Standards APIs.
59+
- 📚 **Fully Typed and Documented**: Fully typed and documented with [TypeDoc](https://typedoc.org/).
60+
- 🚀 **Using WebAssembly for High Performance**: WebAssembly is used for high performance parsing. (_Experimental_)
61+
- 📦 WebAssembly is used for high performance parsing.
5862

5963
## Installation 📥
6064

@@ -276,6 +280,41 @@ ideal for developers looking for in-depth control and flexibility.
276280
- **`class RecordAssemblerTransformer`**: [📑](https://kamiazya.github.io/web-csv-toolbox/classes/RecordAssemblerTransformer.html)
277281
- Handles the assembly of parsed data into records.
278282

283+
### Experimental APIs 🧪
284+
285+
These APIs are experimental and may change in the future.
286+
287+
#### Parsing using WebAssembly for high performance.
288+
289+
You can use WebAssembly to parse CSV data for high performance.
290+
291+
- Parsing with WebAssembly is faster than parsing with JavaScript,
292+
but it takes time to load the WebAssembly module.
293+
- Supports only UTF-8 encoding csv data.
294+
- Demiliter characters are limited to single-byte characters.
295+
- Quotation characters is only `"`. (Double quotation mark)
296+
- If you pass a different character, it will throw an error.
297+
298+
```ts
299+
import { loadWASM, parseStringWASM } from "web-csv-toolbox";
300+
301+
// load WebAssembly module
302+
await loadWASM();
303+
304+
const csv = "a,b,c\n1,2,3";
305+
306+
// parse CSV string
307+
const result = parseStringToArraySyncWASM(csv);
308+
console.log(result);
309+
// Prints:
310+
// [{ a: "1", b: "2", c: "3" }]
311+
```
312+
313+
- **`function loadWASM(): Promise<void>`**: [📑](https://kamiazya.github.io/web-csv-toolbox/functions/loadWASM.html)
314+
- Loads the WebAssembly module.
315+
- **`function parseStringToArraySyncWASM(string[, options]): CSVRecord[]`**: [📑](https://kamiazya.github.io/web-csv-toolbox/functions/parseStringToArraySyncWASM.html)
316+
- Parses CSV strings into an array of records.
317+
279318
## Options Configuration 🛠️
280319

281320
### Common Options ⚙️

0 commit comments

Comments
 (0)