Linux

Linux Work

Comparing two text files and removing duplicates from one

By on Tuesday, January 25, 2011

I had a requirement to compare two files today and remove the entries from the larger list that matched the entries in the smaller list – think a poor man’s mailing list management.
Thanks to the post at StackOverflow, I was able to very quickly remove the entries with the following.
grep -Fiv -f potentialduplicates.txt %lt; fulllist.txt > noduplicates.txt

The flags are as follows –
-F no regexps (fast)
-i case-insensitive
-v invert results
-f get patterns from file

This worked really well and the end user was pleased. I did convert all the entries into all lower case first in excel using =lowercase(a1) and dragging down the list. Copying the new list to a text file meant I had some clean lists to process.
Unfortunately I didn’t have a copy of grep on my Windows7 machine, so I just uploaded it to a linux server to do the processing – quicker than obtaining grep.
As it turns out, I could have used findstr which comes with Windows. The same output can be obtained with
findstr /g:potentialduplicates.txt fulllist.txt >noduplicates.txt

Continue Reading
3
Firewalls Linux Networking VPN

Peer is not authorized to use remote address and Cyberguard firewalls

By on Wednesday, August 22, 2007

One of my customers has a cyberguard firewall that was working perfectly when I configured it in the office. However on shipping it to the customer, it was placed behind a horrible Verizon modem that broke most of the configuration parameters I had put in place. Most of them were fixable straight away, but I’ve never been able to get it to vpn back to the head office. I was getting the error message “Peer is not authorized to use remote address” in the log files.
Various responses in google mentioned I had to add noauth to the /etc/ppp file but this file doesn’t exist on the cyberguard unit. However, using the Advanced section and editing the “pptp.connect to office” file, adding “noauth” (without quotes) to the end of the file, the vpn connection worked.
This isn’t documented anywhere in google that I’ve seen so hopefully this will help someone else.

Continue Reading
0
Linux Ubuntu Wireless

Ubuntu 5.10 is worse!

By on Saturday, September 17, 2005

(in my opinion) than the 5.04 release that was out a while back. At least 5.04 would boot on my machine but 5.10 doesn’t. It gets as far as detecting hot plug devices and then the machine hangs. Various messages are displayed in the other consoles (ctrl-f2 to f4 i think) but nothing obvious as to what is going on. I thought I would try 5.10 on the desktop as the 5.04 would not let me use ndiswrapper for the Hawking Technology wifi usb card that I have (which incidentally uses the zydas 1211 driver). I think that if I risked putting ubuntu onto the hard disk rather then running from the livecd then I might stand a chance of getting it to work, but ideally I want a combination of livecd & usb wifi adapter which means I can use this at other wifi enabled pc’s.

Continue Reading
1
Linux Microsoft Ubuntu Virtual PC

Ubuntu under Virtual PC

By on Wednesday, July 27, 2005

I downloaded Ubuntu this afternoon and tried it out as a virtual machine. Unfortunately the screen was garbled as the default installation uses 24bits for the screen resolution andVirtual PC can’t cope with that.
It can be fixed (each time you boot the virtual pc) by doing the following that I have added to the Ubuntu faq

  • Press Ctrl Alt and Backspace at the same time. This will kill the xserver and drop you down to the console.
  • cd /etc/X11
  • sudo vim xorg.conf
  • Now, scroll down to Section “Screen” and find the entry named “!DefaultDepth”. Change the setting you find there from 24 to 16.
  • Move the cursor to the “2” in 24.
  • Hit the D key. Spacebar key. (You now deleted the “2”.)
  • Hit the D key. Spacebar key. (You now deleted the “4”.)
  • Hit the I key for insert. Type “16”. Hit Ctrl-C to stop inserting.
  • Hit a colon, cursor jumps to bottom left. Type a “w” as the command and press Enter. This saves your changes.
  • Hit a colon, type a “q” as command and press Enter. You are now back at shell prompt.
  • startx to get back to the xserver screen.

Note that you will have to do this each time you boot from your livecd.

Continue Reading
0