Shawson Lim

搬运工. 砌匠. 包工头.

Category
 » mysql
 » golang
 » linux
 » git
 » php
 » front
 » staticstics
 » default
 » tools

[Bash] 多纬数据去重

27 Dec 2014 » linux

需求:从原日志中提供game_id, user_id, ip,并保证在game_id和user_id一至的情况下只保存一条记录。

$ cat a1.log 
100029255 267006098 117.177.110.251 
100029255 632432655 221.227.208.97 
100031999 872069336 119.124.213.102 
100031999 872069335 119.124.213.102 
100031999 872069335 119.124.213.101 
$ cat a2.log 
100028337 558990503 60.180.229.137 
100029249 365790727 211.161.29.213 
100029249 602831481 117.156.5.54 
100029255 1436504718 123.7.116.114 
100029255 802209877 182.145.77.147
$ sort -u a2.log a1.log | wc -l 
10 

$ sort -u -k 1,2 a2.log a1.log | wc -l 
9 
$ sort -u -k 1,2 a2.log a1.log 
100028337 558990503 60.180.229.137 
100029249 365790727 211.161.29.213 
100029249 602831481 117.156.5.54 
100029255 1436504718 123.7.116.114 
100029255 267006098 117.177.110.251 
100029255 632432655 221.227.208.97 
100029255 802209877 182.145.77.147 
100031999 872069335 119.124.213.102 
100031999 872069336 119.124.213.102
© Shawson Lim - https://github.com/linsir123 - Powered by Jekyll.