02 Regular Expression - Regex Replace all characters | regex replace | regex remove | regex alphanumeric with special characters #LinuxTopic

Regular Expression - Regex Replace all characters,  regex replace

Regular Expression - Regex Replace all characters,  regex replace,  regex remove, regex only between, sed replace, sed remove, sed search and replace, how to regex replace all characters, linux command, linux basic, 

In this topic we will regex replace all characters between double quote, before we start i would like to refer our preview topic of regular expression, click here..

Regular Expression - PART 1 click here.. 

Regular Expression - PART 2

OS / Tools / Command

1 - Linux

2 - Bash

3 - grep

4 - sed

5 - logs file ( json logs )

Step 1:

Copy the logs file, which  content a personal information for removing, in this tutorial we have log file in blow directory and file name

log file path :  /var/log/event.log

cp -rv /var/log/event.log /tmp/

Video Tutorial - Regular Expression 

Video Tutorial - Regular Expression  - PART 1 

review logs file and noted which value you want to remove 

cat /var/log/event.log

In the above screenshot we have some PI Field and we will remove below values, like

• accountNumber
• operator 
• custName 
• custID
• cardHolder 
• card 
• Phone 
• Address

For example we will pick one value like accountNumber, in this field we have only numeric value / digit value / number value, 

"accountNumber":"350211212093"

First we will grep  only  "accountNumber" and count it using below command

grep -o '"accountNumber"' /tmp/event.log

grep -o '"accountNumber"' /tmp/event.log | wc -l

We have only 3 count as of now but it can be 100, 1000, 10000 depends on logs, now we print the accountNumber value using below command

grep -o '"accountNumber":"[^"]*' /tmp/event.log

Explanation :

We used a grep command, 

-o for print only matched value, 

We used single quote ( ' ) to grep all because we are including a double quote in grep, 

accountNumber is keyword for match only particular field

Regular Expression - 

[^"]*

[ =  Regex start 

^ = Start of string or start of line depending on multiline mode. so here we started with double quote ^"

] =  End regular expression

* = The * (zero or more) is "greedy"

In the above example we printed the value of account number using the [^"]* regex

Now we will use sed command to replace account numbe, first we will execute  below sed command 

as dry run and validate the replace value.

sed 's/'\"accountNumber\":\"[^\"]*\"'/accountNumbter\":\"REMOVED/g' /tmp/event.log

It looks good now we can replace accound number using below sed command

sed -i 's/'\"accountNumber\":\"[^\"]*\"'/accountNumbter\":\"REMOVED/g' /tmp/event.log

New we have to replace only accountNumber to other field like card in sed command, below

 

sed 's/'\"card\":\"[^\"]*\"'/card\":\"REMOVED/g' /tmp/event.log

To replace card value :

sed -i 's/'\"card\":\"[^\"]*\"'/card\":\"REMOVED/g' /tmp/event.log

Now we can repeat above steps on all fields, i would like to create a script instead manually execution of above command

Script:

1 - Create a script file as you wish with below content

vi /tmp/mask.sh

# In the script we create a backup first in /tmp directory, backup path could be change and then we used a "for loop" using a sed command

#!/bin/bash
cp -rv /var/log/event.log /tmp/event.log

for i in accountNumber custID Address operator custName cardHolder card Phone Address

do

    sed -i 's/'\"$i\":\"[^\"]*\"/$i\":\"REMOVED\"'/g' /tmp/event.log

done

Thanks you !!

I hope this topic gave you all the information you needed. If you have any further questions or would like more detailed directions feel free to contact us using any of the following sources. We look forward to talking to you.