15个实用的数据处理脚本Python与Shell# 15个实用的数据处理脚本Python与Shell高效结合## 1. 批量文件重命名脚本Pythonpythonimport osdef batch_rename(path, old_str, new_str):for file in os.listdir(path):if old_str in file:new_file file.replace(old_str, new_str)os.rename(os.path.join(path, file),os.path.join(path, new_file))print(fRenamed: {file} - {new_file})# 示例将目录下所有包含old_“的文件改为new_”batch_rename(“/path/to/files”, “old_”, “new_”)## 2. 文本文件编码转换Shellbash#!/bin/bash# 将GB2312编码转换为UTF-8for file in *.txt; doiconv -f GB2312 -t UTF-8 “f i l e file file{file%.txt}_utf8.txt”echo “Converted $file to UTF-8”done## 3. 数据去重脚本Pythonpythondef remove_duplicates(input_file, output_file):seen set()with open(input_file, ‘r’) as fin, open(output_file, ‘w’) as fout:for line in fin:if line not in seen:seen.add(line)fout.write(line)# 示例使用remove_duplicates(‘data.txt’, ‘dedup_data.txt’)## 4. 文件批量下载Shellbash#!/bin/bash# 从URL列表下载文件url_list“urls.txt”download_dir“downloads”mkdir -p “KaTeX parse error: Undefined control sequence: \- at position 34: …hile IFS read \̲-̲r url; do …download_dir” “u r l d o n e url done urldoneurl_list”## 5. JSON转CSV工具Pythonpythonimport jsonimport csvdef json_to_csv(json_file, csv_file):with open(json_file) as f:data json.load(f)with open(csv_file, ‘w’, newline‘’) as f:writer csv.DictWriter(f, fieldnamesdata[0].keys())writer.writeheader()writer.writerows(data)# 示例json_to_csv(‘data.json’, ‘data.csv’)限于篇幅后续脚本只展示核心代码片段## 6. 日志文件时间筛选Shellbashgrep -E “2023-07-[0-9]{2}” access.log july_logs.txt## 7. 图片批量压缩Pythonpythonfrom PIL import Imageimport osdef compress_images(input_dir, output_dir, quality85):for file in os.listdir(input_dir):if file.endswith((‘.jpg’, ‘.jpeg’, ‘.png’)):img Image.open(os.path.join(input_dir, file))img.save(os.path.join(output_dir, file), qualityquality)## 8. 系统进程监控Shellbash#!/bin/bash# 监控CPU使用率超过50%的进程top -b -n1 | awk ‘NR7 $9 50 {print $1 “\t” $9 “\t” $12}’## 9. Excel数据合并Pythonpythonimport pandas as pddef merge_excels(file_pattern, output_file):all_data pd.DataFrame()for file in glob.glob(file_pattern):df pd.read_excel(file)all_data pd.concat([all_data, df])all_data.to_excel(output_file, indexFalse)## 10. 批量SSH命令执行Shellbash#!/bin/bash# 在多台服务器上执行相同命令hosts(“server1” “server2” “server3”)command“df -h”for host in “${hosts[]}”; doecho “h o s t s s h host ssh hostsshhost” “$command”done## 11. 数据库备份脚本Shellbash#!/bin/bash# MySQL数据库备份DB_USER“user”DB_PASS“password”DB_NAME“database”BACKUP_DIR“/backups”mysqldump -uKaTeX parse error: Undefined control sequence: \- at position 10: DB\_USER \̲-̲pDB_PASSD B _ N A M E ∣ g z i p DB\_NAME | gzip DB_NAME∣gzipBACKUP_DIR/db_$(date %Y%m%d).sql.gz## 12. 网页数据抓取Pythonpythonimport requestsfrom bs4 import BeautifulSoupdef scrape_links(url, selector):r requests.get(url)soup BeautifulSoup(r.text, ‘html.parser’)return [a[‘href’] for a in soup.select(selector)]## 13. 磁盘空间告警Shellbash#!/bin/bashthreshold80df -h | awk -v t$threshold ‘NR1 {gsub(/%/,“”); if ($5 t) print $1 is $5 “% full”}’## 14. CSV数据清洗Pythonpythonimport pandas as pddef clean_csv(input_file, output_file):df pd.read_csv(input_file)# 填充空值df.fillna(method‘ffill’, inplaceTrue)# 去除重复行df.drop_duplicates(inplaceTrue)df.to_csv(output_file, indexFalse)## 15. 自动打包部署Shellbash#!/bin/bash# 自动构建并部署项目project_dir“/home/user/project”cd “$project_dir” \git pull \npm install \npm run build \rsync -avz dist/ deployserver:/var/www/html/## 结语这些脚本覆盖了日常数据处理中的常见需求可以根据实际情况进行调整。Python适合复杂的数据处理任务而Shell则在系统管理方面更胜一筹。两种语言结合使用能大幅提升工作效率。
