winget install --id=jayhuang75.med -e
A simple enough CLI tool with auditable capability for Data Masking/Encyption/Decryption for CSV/JSON files.
M.E.D. (Mask, Encrypt, Decrypt) is a Rust-powered command-line tool designed for efficient data masking, encryption, and decryption of CSV and JSON files. Built with performance in mind, it offers robust capabilities to handle large datasets securely while maintaining minimal memory consumption.
Key Features:
Audience & Benefit: Ideal for data engineers, analysts, and security professionals who need to handle sensitive data securely. M.E.D. provides a lightweight yet powerful solution to protect privacy while maintaining performance, making it suitable for large-scale operations without compromising efficiency.
Installable via winget, M.E.D. is designed to be accessible and efficient in any environment where secure data handling is essential.
This is a personal hobby project; based on the observation, sometimes we need a simple enough CLI tool with auditable capability for Data Masking/Encryption/Decryption for CSV/JSON files.
There are two central crates in this package.
The binary name for M.E.D. is med; it depends on the med_core.
Archives of precompiled binaries for med are available for Windows, macOS, and Linux. Users of platforms not explicitly mentioned below are advised to download one of these archives.
dnf install med
brew tap jayhuang75/med
brew install med
Model Name: MacBook Pro
Processor Name: 6-Core Intel Core i7
Processor Speed: 2.6 GHz
Total Number of Cores: 6
Memory: 16 GB
File Type | Records | File Size | File Counts | Mode | Field num for mask/encrypt | Elapsed Time | Memory Consumption |
---|---|---|---|---|---|---|---|
CSV | 120,000,000 | 2.8G | 3 | mask | 1 | ~78 seconds (1.3 mins) | ~2 MB |
CSV | 120,000,000 | 2.8G | 3 | encrypt (DES64) | 1 | ~182 seconds (3 mins) | ~1.9 MB |
CSV | 120,000,000 | 2.8G | 3 | encrypt (AES128) | 1 | ~221 seconds (3.6 mins) | ~1.9 MB |
CSV | 20,000,000 | 471M | 2 | mask | 1 | ~7 seconds | ~1.8 MB |
CSV | 10,000,000 | 236M | 1 | mask | 1 | ~5 seconds | ~1.8 MB |
JSON | 129,220 | 10G | 129,220 | mask | 1 | ~2200 seconds(36 mins) | ~62 MB |
JSON | 64,641 | 5.1G | 64,641 | mask | 1 | ~792 seconds(13 mins) | ~30 MB |
JSON | 23,519 | 1.9G | 23,519 | mask | 1 | ~284 seconds(4.7 mins) | ~18 MB |
JSON | 23,519 | 1.9G | 23,519 | encrypt | 1 | ~282 seconds(4.4 mins) | ~18 MB |
This project follows the all-contributors specification.
Please complete the template before the PR.
Contributions of any kind are welcome!
Give a ⭐️ if this project helped you!
Copyright © 2023 Wei Huang
This project is Licensed under Apache.