业务背景约定五天前的HDFS数据为过期版本数据,,写一个awk脚本自动删除过期版本数据
$ Hadoop fs -ls /user/pms/workspace/ouyangyewei/dataFound 9 itemsdrwxr-xr-x – pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-01drwxr-xr-x – pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-02drwxr-xr-x – pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-03drwxr-xr-x – pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-04drwxr-xr-x – pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-05drwxr-xr-x – pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-06drwxr-xr-x – pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-07drwxr-xr-x – pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-08drwxr-xr-x – pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-09脚本实现# ———————————————————## 删除历史版本(五天前的为过期版本数据)## ———————————————————
old_version=$(hadoop fs -ls /user/pms/workspace/ouyangyewei/data | awk ‘BEGIN{ five_days_ago=strftime(“%F”, systime()-5*24*3600) }{ split($8,arr,”http://www.linuxidc.com/”); if(arr[7]<five_days_ago){printf “%s\n”, $8} }’)arr=(${old_version// / })for version in ${arr[@]} do hadoop fs -rmr $versiondone
执行以后
$ hadoop fs -ls /user/pms/workspace/ouyangyewei/dataFound 4 itemsdrwxr-xr-x – pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-06drwxr-xr-x – pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-07drwxr-xr-x – pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-08drwxr-xr-x – pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-09
awk赋值操作符
awk转义序列与算术操作符
AWK简介及使用实例
AWK 简介和例子
Shell脚本之AWK文本编辑器语法
正则表达式中AWK的学习和使用
文本数据处理之AWK 图解
如何在Linux中使用awk命令
文本分析工具-awk
本文永久更新链接地址:
爱情要完结的时候自会完结,到时候,你不想画上句号也不行。