VBScript: Recursively Search and Replace in Windows

Recently I needed a command-line tool to perform recursive search and replacement inside text files located in a directory and its subfolders under Windows platforms. Several Open Source and proprietary tools exist but I ended by writing my own VBScript for simplicity, customization and portability reasons.

Command-line:


{CScript [/nologo]|WScript} /Path/to/SearchReplaceInFiles.vbs {Folder Path} {String} {ReplaceStringWith} {*.Extension}

Where:

Folder Path: Folder and its subfolders to find files
String: The string to be searched
ReplaceStringWith: The replacement string
*.Extension: File(s) extension. It can be *.*, *.txt …

Below a sample command-line:


C:\> CScript /nologo C:\Briolidz\Dev\Tools\SearchReplaceInFiles.vbs C:\Briolidz\Dev "My Word 1" "My Word 2" *.txt

SearchReplaceInFiles.vbs forces to run in CScript to prevent a separate popup for each WScript.Echo line.

SearchReplaceInFiles.vbs source code:

'
' Recursive search and replacement inside text files located in a directory and
' its subfolders under Windows platforms.
'
' {CScript [/nologo]|WScript} /Path/to/SearchReplaceInFiles.vbs {Folder Path} {String} {ReplaceStringWith} {*.Extension}
'

' Force script to run in CScript to avoid WScript.Echo messages pop-up box
ForceCScript()

Dim Args, FolderPath, FindString, ReplaceStringWith, FSO

' Check arguments
Args = WScript.Arguments.Count
If Args <> 4 then
 WScript.Echo
 WScript.Echo "Usage : {CScript [/nologo]|WScript} /Path/to/SearchReplaceInFiles.vbs {Folder Path} {String} {ReplaceStringWith} {*.Extension}"
 WScript.Echo
 WScript.Echo " Folder Path: Folder and its subfolders to find files"
 WScript.Echo " String: The string to be searched"
 WScript.Echo " ReplaceStringWith: The replacement string"
 WScript.Echo " *.Extension: file(s) extension. Can be *.*, *.txt ..."
 WScript.Echo
 WScript.Quit(1)
end If

' Get arguments
FolderPath = WScript.Arguments(0)
FindString = WScript.Arguments(1)
ReplaceStringWith = WScript.Arguments(2)
Extension = WScript.Arguments(3)

' Creating File System Object
Set FSO = CreateObject("Scripting.FileSystemObject")

' Check extension format
If Not Left(Extension, 2) = "*." Then
 WScript.Echo("Invalid extension " & Extension)
 WScript.Quit(1)
End If

' Check destionation folder
If Not FSO.FolderExists(FolderPath) Then
 WScript.Echo("Folder " & FolderPath & " does not exist !")
 WScript.Quit(1)
End If

' Files extension
Extension = LCase(Mid(Extension, 3))

' Call the GetFiles function to get all files
WScript.Echo
WScript.Echo("Searching for *." & Extension & " files in " & FolderPath & " folder ...")
GetFiles FolderPath, Extension
WScript.Echo("Completed")

Set FSO = Nothing

Function ForceCScript()
 Dim StrArg, Str

 If Not LCase(Right(WScript.FullName, 12)) = "\cscript.exe" Then
  For Each StrArg In WScript.Arguments
   If InStr(StrArg, " ") Then StrArg = """" & StrArg & """"
   Str = Str & " " & StrArg
  Next

  StrCmd = "CScript //nologo """ & WScript.ScriptFullName & """" & Str
  Set WshShell = CreateObject("WScript.Shell")

  ' Rerun the script in CScript
  Set ObjExec = WshShell.Exec(StrCmd)

  ' Wait until the script exits
  Do While ObjExec.Status = 0
   WScript.Sleep 100
  Loop

  ' Exit with CScript's return code
  WScript.Quit ObjExec.ExitCode

  Set WshShell = Nothing
  Set ObjExec = Nothing
 End If
End Function

Function GetFiles(FolderPath, Extension)
 On Error Resume Next

 Dim ObjFolder, ObjSubFolders
 Dim ObjFiles, ObjFile

 Set ObjFolder = FSO.GetFolder(FolderPath)
 Set ObjFiles = ObjFolder.Files

 ' Write all files to output files
 For Each ObjFile In ObjFiles
  StrExtension = LCase(FSO.GetExtensionName(ObjFile))

  If Extension = "*" Or StrExtension = Extension Then
   ' Read source text file
   FileContents = GetFile(ObjFile)

   ' Replace all string In the source file
   NewFileContents = Replace(FileContents, FindString, ReplaceStringWith, 1, -1, 1)
   If NewFileContents <> FileContents Then
    ' Write result If different
    WScript.Echo(" Updating File: " & ObjFile.Path)
    WriteFile ObjFile, NewFileContents
   End If
  End If
 Next

 ' Getting all subfolders
 Set ObjSubFolders = ObjFolder.SubFolders

 ' Getting all Files from subfolder
 For Each ObjFolder In ObjSubFolders
  GetFiles ObjFolder.Path, Extension
 Next
End Function

' Read text file
function GetFile(FileName)
 If FileName <> "" Then
  Dim FS, FileStream

  Set FS = CreateObject("Scripting.FileSystemObject")

  on error resume Next
  Set FileStream = FS.OpenTextFile(FileName)
  GetFile = FileStream.ReadAll
 End If
End Function

' Write string As a text file
function WriteFile(FileName, Contents)
 Dim OutStream, FS

 on error resume Next
 Set FS = CreateObject("Scripting.FileSystemObject")
 Set OutStream = FS.OpenTextFile(FileName, 2, True)
 OutStream.Write Contents
End Function
Advertisements

Windows driver debugging with WinDbg and VMWare

Virtualization Software such as VMware Workstation enables driver and kernel-mode code developers to speed up development, debugging and testing resulting in faster time to deployment. Snapshots provide a fast and easy way to revert the virtual machine to a previous state. This feature greatly simplifies recreation of specific states or conditions to troubleshoot problems and system crashes.

WinDbg is a debugging tool from Microsoft for user and kernel mode debugging. WinDbg is a GUI interface and a console interface along with some debugging extensions. Using virtual machines, WinDbg can be used to debug kernel code without the need for two physical computers.

This tutorial shows how to debug a simple Windows driver running inside a VMware virtual machine with WinDbg using a single physical machine. The following typical debugging scenarios are covered:

    • Debugging host’s virtual machine: WinDbg runs inside a physical computer to debug a virtual machine.
    • Debugging between two virtual machines: WinDbg runs inside a virtual machine to debug the second one.

Basic familiarity with device driver development and kernel debugging is assumed.

Prerequisites

The working folder is C:\Briolidz\Dev\BriolidzSampDrv.

I assume that your host and guests root system partition is C:\. The kernel debugging setup will use a virtual serial port on COM2 at 115200 bps. The custom named pipe is called BriolidzDbgPipe.

These parameters can be changed to fit your own setup.

VMware machine configuration

In the first debugging scenario, you will debug a virtual machine directly over a named pipe from the host operating system. In the second scenario, you will connect two virtual machines to the same named pipe by creating a virtual null-modem cable. The following configuration steps can be easily adapted for other virtualization software such as VirtualBox and Virtual PC.

In both scenarios, you need to setup the guest virtual machine to debug as follows:

1. Powered off the virtual machine.
2. Open VM then select Settings in VMware Workstation menu.
3. In the VMware Machine Settings dialog box, click Add.
4. In the Add Hardware Wizard dialog box, select Serial Port and click Next.
5. On next page, select Output to named pipe and click Next.
6. Set Name pipe to \\.\pipe\BriolidzDbgPipe. Make sure you select This end is the server and The other end is virtual machine. Check Connect at power on then click Finish.
7. After clicking Finish, select the newly created serial port and check Yield CPU on poll.

You should have something similar to the following screenshot. Note that new serial port has number 2 which corresponds to COM2. If you get assigned another number then make sure to replace COM2 with COM<your assigned number> in the Virtual machine configuration section.

Skip this step, if you are going to use WinDbg on your host (physical computer).

So you decided to use a second virtual machine as a development and debugging machine to replace the host physical machine. In this case, do:

1. Powered off the virtual machine.
2. Open VM then select Settings in VMware Workstation menu.
3. In the VMware Machine Settings dialog box, click Add.
4. In the Add Hardware Wizard dialog box, select Serial Port and click Next.
5. On next page, select Output to named pipe and click Next.
6. Set Name pipe to \\.\pipe\BriolidzDbgPipe. Make sure you select This end is the client and The other end is virtual machine. Check Connect at power on then click Finish.
7. After clicking Finish, select the newly created serial port and check Yield CPU on poll.

You should have something similar to the following screenshot. Note that the new serial port has number 2 which corresponds to COM2. As stated earlier, if you get assigned another number then replace COM2 with COM<your number> in the Attaching WinDBG Debugger section.

Virtual machine configuration

To enable kernel debugging, you need to change Windows boot parameters. Your virtual machine will also be configured to use serial debugging on COM2 at 115200 bps. Power on your virtual machine, log in then apply the following modifications according to your operating system version.

1. Windows XP

Windows XP uses a configuration file called boot.ini on the root of the system partition (generally the C:\ drive) to control how the operating system is booted and any startup options.

The boot.ini file has the Hidden, System, and Read-Only attributes set by default. Open a command prompt, and change them:

attrib -s -h -r C:\boot.ini

The boot.ini file is a standard ASCII text editor. Double click it to edit and duplicate the matching default entry defined in boot loader section, change its title by adding [DEBUG] for instance, add /Debug, /debugport=com2 and /baudrate=115200 switches. Below, a modified sample for Windows XP professional:

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional [DEBUG]" /noexecute=optin /fastdetect /Debug /debugport=com2 /baudrate=115200
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /noexecute=optin /fastdetect

You can reduce the timeout value in order to speed up booting.

Restore the Read-Only, Hidden, and System attributes of the boot.ini file, type the following at a command prompt:

attrib +h +r +s C:\boot.ini

2. Windows Vista and 7

Windows Vista and 7 use the bcdedit command line tool to configure the boot menu. Start a command prompt with administrator privileges and run the following commands:

bcdedit /set {current} debug yes
bcdedit /set {current} debugtype serial
bcdedit /set {current} debugport 2
bcdedit /set {current} baudrate 115200

Set the operating system selection menu to be displayed when booting and loading the operating system.

bcdedit /set {bootmgr} displaybootmenu yes

You can reduce boot timeout as follows:

bcdedit /timeout 10

To view the current configuration, run:

bcdedit

Sample driver code

Depending on your debugging scenario, log in to your development machine: the host (the physical computer) or the virtual machine. You will write the necessary files to create a do-nothing sample driver called BriolidzSampDrv.sys. This driver has minimal DriverEntry and DriverUnload routines and will write some debug message.

In the working folder, create a file called BriolidzSampDrv.c containing the following source code:

#include <wdm.h>
 
void DriverUnload(PDRIVER_OBJECT pDriverObject)
{
    DbgPrint("BriolidzSampDrv: Driver unloading.\n");
}

NTSTATUS DriverEntry(PDRIVER_OBJECT DriverObject, PUNICODE_STRING RegistryPath)
{
    DriverObject->DriverUnload = DriverUnload;
	
    DbgPrint("BriolidzSampDrv: DriverEntry.\n");
	
    return STATUS_SUCCESS;
}

There are two additional files required to build a device driver: sources and makefile.

Create sources file containing the following lines:

TARGETNAME=BriolidzSampDrv
TARGETTYPE=DRIVER
SOURCES=BriolidzSampDrv.c

The makefile file only needs to contain this line:

!INCLUDE $(NTMAKEENV)\makefile.def

To install the sample driver, you will use a basic method based on a registry file. Create InstallSampDrv.reg file containing the following lines:

Windows Registry Editor Version 5.00
  
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BriolidzSampDrv]
"Type"=dword:00000001
"Start"=dword:00000000
"ErrorControl"=dword:00000001
"Group"="Base"
"ImagePath"="\\SystemRoot\\System32\\Drivers\\BriolidzSampDrv.sys"
"Description"="Briolidz - Sample Driver"
"DisplayName"="BriolidzSampDrv"

Building the sample driver

Download the Windows Driver Kit. Run the installer and make sure to select at least Build Environments and Debugging Tools for Windows. This will install WinDbg and the necessary environments to build drivers.

When writing this tutorial, I used WDK version is 7.1.0 which include WinDbg version 6.12.2.633. Under Windows 7 professional 64-bit, the default installation folder is C:\WinDDK\7600.16385.1 and the WDk’s start menu group is called Windows Driver Kits. WinDbg is installed in C:\WinDDK\7600.16385.1\Debuggers folder. In case you are using a different WDK version, you will just have to update paths in the following sections.

From the Windows Driver Kits start menu group, launch the command-line build environment for the desired target platform and architecture. Since The BriolidzSampDrv sample driver does not use platform specific code, you can safely use the x86 Checked Build Environment and x64 Checked Build Environment under Winows 7 submenu.

For both target platforms, open the command-line build environment, change to the working directory containing the sample driver source code and enter the command:

C:\Briolidz\Dev\BriolidzSampDrv> build

The 64-bit driver is built in C:\Briolidz\Dev\BriolidzSampDrv\objchk_win7_amd64\amd64\BriolidzSampDrv.sys and the 32-bit driver is built in C:\Briolidz\Dev\BriolidzSampDrv\objchk_win7_x86\i386\BriolidzSampDrv.sys.

Sample driver installation

Reboot the guest virtual machine to debug. When the boot menu is displayed, always choose the menu item containing [debugger enabled].

If you are planning to debug the x64 driver version on Windows Vista/7 (necessarily x64), then press F8 key and select Disable Driver Signature Enforcement from the menu. This operation must be done at every boot because 64-bit Windows systems will not allow drivers to be loaded unless they have a valid digital signature.

If you cannot digitally sign your driver, you can test-sign it and enable the TESTSIGNING mode using the bcdedit command to avoid manually disabling driver signature procedure at every boot time.

After logging in a session with administrator privileges, copy the BriolidzSampDrv.sys (from your working folder) to C:\Windows\System32\drivers. Also copy the InstallSampDrv.reg file to a location of your choice then double-click it to install the driver.

Changes will take effect after the next reboot.

Attaching WinDBG Debugger

At this point your debugging platform is ready to be used. These are the main steps to follow:

1. The WinDbg command line gives you the ability to use environment variables in order to create workspaces that contain your custom debugging settings.

In the following, if you are debugging the 32-bit driver then replace C:\Briolidz\Dev\BriolidzSampDrv\objchk_win7_amd64\amd64 with C:\Briolidz\Dev\BriolidzSampDrv\objchk_win7_x86\i386.

Inside the development machine (host or virtual machine), open a command prompt and type:

set _NT_EXECUTABLE_IMAGE_PATH= C:\Briolidz\Dev\BriolidzSampDrv\objchk_win7_amd64\amd64
set _NT_SOURCE_PATH=C:\Briolidz\Dev\BriolidzSampDrv

If you chose to debug a virtual machine from your host (physical computer):

"C:\WinDDK\7600.16385.1\Debuggers\windbg.exe" -b -k com:pipe,port=\\.\pipe\BriolidzDbgPipe,resets=0,reconnect

Otherwise:

"C:\WinDDK\7600.16385.1\Debuggers\windbg.exe" -b -k com:port=com2,resets=0,reconnect

WinDbg GUI will popup and display Waiting to reconnect message.

2. Start the target virtual machine.

3. While booting, WinDbg will halt the target system. In the WinDbg command pane, set a breakpoint in DriverEntry routine as follows:

bu BriolidzSampDrv!DriverEntry

By default, DbgPrint messages do not appear in WinDbg when the driver is running on Windows Vista/7 due to filtering reasons. You can clear this filtering using this simple call:

ed nt!Kd_DEFAULT_Mask 0x8

Press g or F5 key to continue.

After few seconds (or more depending on your computer speed), WinDbg will halt again. You should see something similar to this screenshot.

At this level, you can debug the driver by setting other breakpoints, stepping in the code, monitoring variables and expressions.

Compiling Qt with Visual Studio 2010

Qt is an object oriented C++ cross-platform GUI toolkit which can be used to build applications that run natively on several platforms including Windows, Linux, Mac OS, and embedded Linux without source code changes. Qt is available under GPL v3, LGPL v2 and a commercial license.

This tutorial is intended for Windows users preferring to stick to Visual Studio 2010 as a primary development environment instead of Qt Creator. It provides step-by-step instructions for compiling Qt from source using the open-source distribution.

Prerequisites

    • Visual Studio 2010 Professional with SP1.
    • Windows 7 Professional.
    • Qt 4.8.0 open-source distribution (or a recent Qt 4.7.x version).
    • Qt Visual Studio Add-in.
    • ActivePerl.

Setup Visual Studio 2010

1. Download and install Microsoft Visual Studio 2010 Service Pack 1.

2. For x64 users, it is recommended to download and install this patch for Visual Studio 2010, if it is not already up-to-date.

3. Download and install the Qt Visual Studio Add-In. After installation, a new menu called Qt is added to Visual Studio 2010.

Download and install ActivePerl

ActiveState provides a Perl distribution for Windows platforms. Download and install the latest x86 version from here. Stick to x86 unless you have any special reason for using 64-bit applications.

During installation, make sure that Add Perl to the PATH environment variable and Create Perl file extension association checkboxes are selected.

Download and compile Qt

1. Download the Qt source code as a .zip file. The Qt-everywhere package version 4.8.0 is located here.

2. Unzip the previous file into a location without spaces and special characters. I will use C:\Qt\4.8.0.

3. Open a Visual Studio Command Prompt: All programs -> Microsoft Visual Studio 2010 -> Visual Studio Tools -> Visual Studio Command Prompt (2010). Stick to x86 unless you have any special reason for using 64-bit applications.

4. Get Qt configured:

C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC> cd C:\Qt\4.8.0
C:\Qt\4.8.0> configure -debug-and-release -opensource -shared -platform win32-msvc2010

Type y and press Enter to accept the terms of the license agreement.

You can of course specify additional options to add or remove some features depending upon your requirements. Type configure -help to get more details.

5. Compile Qt:

C:\Qt\4.8.0> nmake

The compilation takes a long time. Be patient.

Create a new Qt project

1. Open Visual Studio 2010. Go to Qt –> Qt Options and add the folder C:\Qt\4.8.0.

 

 

2. Create a new Qt project (File -> New -> Project… then select Qt4 Projects) or open an existing .pro file.

Debug and Release versions of the Qt shared libraries (QtCore4.dll, QtGui4.dll, QtCore4d.dll, QtGui4d.dll …) are located in C:\Qt\4.8.0\bin folder.  You will need to distribute the relevant ones used by the application when deploying on other systems.

Improving Debian 6 (Squeeze) fonts rendering

The default Terminal and desktop fonts rendering of Debian Squeeze are nice but when it comes to intensive web browsing, particularly in high resolution, Debian is not as much comfortable as other operating systems such as Mac OS, Ubuntu and Windows.

The following instructions and commands provide an easy way to improve the overall display rendering with minimal changes to enable bold effects in GNOME sessions for a pleasant Web browsing experience with Google Chrome.

1. Update fonts configuration files.

Switch to root user.

$ su -
Password:

Download the new fontconfig XML files.

# wget http://www.osresources.com/files/centos-windows-fonts/fontconfig.tbz

Backup old configuration files then extract new ones into /etc/fonts.

# for i in `ls /etc/fonts/*.conf`; do cp $i $i.bak; done
# tar xvjpf fontconfig.tbz -C /etc/fonts/

Clean up.

# rm fontconfig.tbz

Exit root session.

# exit

Restart your GNOME session.

$ gnome-session-save --logout

2. Download, install and configure Google Chrome.

Switch to root user.

$ su -
Password:

Debian Squeeze comes with Chromium Web Browser version 6.x. Download and install the latest version of Google Chrome. Replace i386 by amd64 if your operation system is 64 bits.

# wget https://dl.google.com/linux/direct/google-chrome-stable_current_i386.deb
# dpkg -i google-chrome-stable_current_i386.deb

If you get the following message:

Errors were encountered while processing:
  google-chrome-stable

Then install missing dependencies as follows:

# apt-get -f install

Clean up and exit root session.

# rm google-chrome-stable_current_i386.deb
# exit

Run Google Chrome by opening Applications > Internet > Google Chrome or by issuing the following command:

$ google-chrome &

Click on the wrench icon in top right corner and select Preferences. In the Under the Hood tab, set Page zoom to 125%.

Installing VMware Tools on Debian 6 (Squeeze)

VMware Tools are an optional set of drivers and utilities that can greatly improve the speed, efficiency and manageability of the virtual machine’s guest operating system. These tools allow cut and paste, mouse and clock synchronization, folder sharing across environments and several other key benefits.

This tutorial has been tested on Debian Squeeze running on VMware Workstation 8. The following steps should work on all the latest versions of Debian and VMware Workstation/Player.

1. In an X session, launch a Terminal and switch to root user.

$ su -
Password:

2. Update your system.

# apt-get update

3. Install kernel headers and tools required to compile and install VMware Tools.

# apt-get install make gcc linux-headers-$(uname -r)

4. Mount the VMware Tools virtual CD.

In VMware Workstation menu, click VM then select Install VMware Tools… menu item. This will put a VMware Tools CD icon on the desktop and a File Browser window should be opened immediately.

If the file browser doesn’t open or it is empty, then in the VMware Workstation menu, click VM, choose Cancel VMware Tools Installation, accept if asked to disconnect the CD-ROM and try again this step.

5. Extract VMware Tools to the tmp directory.

# cd /media/cdrom
# tar -C /tmp -zxvf VMwareTools-*.tar.gz

6. Start the installation.

# cd /tmp/vmware-tools-distrib
# ./vmware-install.pl

Accept the defaults for every question.

7. Clean up.

# cd /tmp
# rm -rf vmware-tools.distrib

8. VMware Tools is now installed and configured. Restart the machine to get all features working.

# reboot

Building embedded ARM systems with Crosstool-NG

This tutorial deals with creating a minimal embedded ARM system from scratch. The approach taken here requires no additional hardware beyond the development host and the source code of the required softwares. QEMU is used to run and debug the ARM system.

This tutorial details the following topics:

    • Building a toolchain for ARM using Crosstool-NG.
    • Building QEMU with ARM system emulation support.
    • Building an ARM Linux kernel.
    • Building and extending a tiny filesystem from scratch, based on BusyBox.
    • Debugging ARM applications inside QEMU.

I intentionally kept the default settings when applicable for simplicity and clarity reasons. Many things can be tweaked; you are encouraged to try alternate ways to become familiar with embedded systems.

Prerequisites

When writing this tutorial, I used the following environment:

Created a working folder:

$ mkdir -p ~/dev-embedded/ctool-ng

Building Crosstool-NG

Crosstool-NG allows building cross-compiling toolchains for multiple target architectures. This versatile tool provides a menuconfig configuration interface, which makes it quite easy to use. Crosstool-NG comes with a set of ready-made configuration files for various typical setups and supports glibc, uclibc and eglibc.

Install needed packages:

$ sudo apt-get install build-essential libncurses5-dev
$ sudo apt-get install automake libtool bison flex texinfo
$ sudo apt-get install gawk curl cvs subversion gcj-jdk
$ sudo apt-get install libexpat1-dev python-dev

Download, configure and install Crosstool-NG locally using —local option (Note that is possible to install it globally):

$ cd ~/dev-embedded/ctool-ng
$ wget http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.13.2.tar.bz2
$ tar xjf crosstool-ng-1.13.2.tar.bz2
$ cd crosstool-ng-1.13.2
$ ./configure --local
$ make
$ make install

Crosstool-NG comes with a set of ready-made configuration files for various typical setups called samples. They can be listed by running:

$ ./ct-ng list-samples

Use the ARM EABI cross compiler for use on linux. Configure and produce this toolchain by issuing:

$ ./ct-ng arm-unknown-linux-gnueabi

Refine your configuration by running the menuconfig interface:

$ ./ct-ng menuconfig

At this point it is up to you to select what options you want. You can explore the available options by traveling through the menus and looking at the help for some of the options.

I recommend the following minimal change:

In Path and misc options, set Number of parallel jobs to 2 times the number of CPU cores in your workstation. This causes a faster building of your toolchain.

Disable Fortran and Java programming languages, if you are not going to use them:
In C compiler options, uncheck Fortran and Java.

Build Crosstool-NG:

$ ./ct-ng build

The operation Retrieving needed toolchain components’ tarballs starts. The ./build.log file is useful to track issues. If you get a problem related to fetching a tarball then try downloading the same version from another location or mirror into ./.build/tarballs folder and rerun the ./ct-ng build command.

Compilation takes a long time, be patient.

For the subsequent steps in this tutorial, you need to add path of executables you just installed to your PATH environment variable by issuing the following command:

$ export PATH=$PATH:$HOME/x-tools/arm-unknown-linux-gnueabi/bin

If you want to make it session wide then add the previous command to ~/.bashrc or ~/.pam_environment.

Now, you can verify that your ARM toolchain works by typing:

$ arm-unknown-linux-gnueabi-gcc -v

Building an ARM Linux kernel

This ARM kernel, built with a predefined configuration for the versatile platform board, will be used in the QEMU virtual machine.

Download and configure the kernel:

$ cd ~/dev-embedded/ctool-ng
$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.2.tar.bz2
$ tar xjf linux-3.2.tar.bz2
$ cd linux-3.2
$ make ARCH=arm versatile_defconfig
$ make ARCH=arm menuconfig

The last command brings up the kernel configuration menu, go to the Kernel Features section, check Use the ARM EABI to compile the kernel then exit (Make sure to save your changes).

Compile the kernel:

$ make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- all

Once it’s done building, the compressed kernel image is located at ./arch/arm/boot/zImage.

Building QEMU

QEMU is a high performance full system simulator supporting both emulation and virtualization. QEMU supports emulating several CPU architectures that are different from the host running the simulation. QEMU is used as the core device model by both Xen and KVM.

Install needed packages:

$ sudo apt-get install zlib1g-dev libsdl-dev

Download, configure and build QEMU from source:

$ cd ~/dev-embedded/ctool-ng
$ wget http://download.savannah.gnu.org/releases/qemu/qemu-0.14.1.tar.gz
$ tar xzf qemu-0.14.1.tar.gz
$ cd qemu-0.14.1
$ ./configure --enable-sdl --disable-kvm --enable-debug --target-list=arm-softmmu
$ make

The executable located at ./qemu-0.14.1/arm-softmmu/qemu-system-arm is the full system emulator for the ARM architecture.

Building GNU Hello

The very simple GNU Hello program, a minimal example of a GNU package that prints Hello, world‘ when you run it, will be debugged in the QEMU virtual machine.

Download, configure and build GNU Hello:

$ cd ~/dev-embedded/ctool-ng
$ wget http://ftp.gnu.org/gnu/hello/hello-2.7.tar.gz
$ tar xzf hello-2.7.tar.gz
$ cd hello-2.7
$ CC=arm-unknown-linux-gnueabi-gcc ./configure --host=arm-unknown-linux-gnueabi
$ make

The ARM-executable is located at ./hello-2.7/src/hello.

Building BusyBox

BusyBox combines tiny versions of many common UNIX utilities into a single small executable, including a shell and an init process. BusyBox is used extensively to build embedded systems of many types and is customizable so that only the needed utilities are built.

Download, configure and build BusyBox from source:

$ cd ~/dev-embedded/ctool-ng
$ wget http://busybox.net/downloads/busybox-1.19.3.tar.bz2
$ tar xjf busybox-1.19.3.tar.bz2
$ cd busybox-1.19.3
$ make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- defconfig
$ make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- install

A folder, containing a minimal root filesystem, called _install is created.

Customizing the filesystem

In order to debug the GNU Hello program, a customization of our minimal filesystem created in the previous section is required. First, copy the following:

  • gdbserver built with arm-unknown-linux-gnueabi.
  • GNU hello executable compiled previously.

Copy required executables:

$ cd ~/dev-embedded/ctool-ng
$ cd busybox-1.19.3/_install
$ CROSSTOOLNG_DIR=$HOME/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/
$ cp $CROSSTOOLNG_DIR/debug-root/usr/bin/gdbserver usr/bin/
$ cp ../../hello-2.7/src/hello usr/bin/

The minimal filesystem still lacks some shared libraries to run busybox, gdbserver and hello programs. These missing libraries can be enumerated using arm-unknown-linux-gnueabi-readelf (created when building the ARM toolchain) as following:

$ arm-unknown-linux-gnueabi-readelf ./usr/bin/hello -a |grep lib
$ arm-unknown-linux-gnueabi-readelf ./usr/bin/gdbserver -a |grep lib
$ arm-unknown-linux-gnueabi-readelf ../busybox -a |grep lib

After analyzing these commands output, you likely just needed to add the following libraries:

$ mkdir -p lib
$ cp $CROSSTOOLNG_DIR/sysroot/lib/ld-linux.so.3 lib/
$ cp $CROSSTOOLNG_DIR/sysroot/lib/libgcc_s.so.1 lib/
$ cp $CROSSTOOLNG_DIR/sysroot/lib/libm.so.6 lib/
$ cp $CROSSTOOLNG_DIR/sysroot/lib/libc.so.6 lib/
$ cp $CROSSTOOLNG_DIR/sysroot/lib/libdl.so.2 lib/

To add networking capabilities to the guest ARM system side and optionally use the same keyboard layout as your host machine, you need to create additional folders in the filesystem:

$ mkdir -p proc sys dev etc etc/init.d

Optionally, create a BusyBox-compatible keymap, in the filesystem, from the currently loaded keymap of your host:

$ sudo busybox dumpkmap > etc/host.kmap

The default initialization script that BusyBox searches for is /etc/init.d/rcS. Instead of using inittab, this is the preferred method to initialize an embedded system based on BusyBox. Create the rcS script as follows:

$ cat > ./etc/init.d/rcS << EOF
#!/bin/sh

# Mount the /proc and /sys filesystems
mount -t proc none /proc
mount -t sysfs none /sys

# Populate /dev
/sbin/mdev -s

# Enable the localhost interface
ifconfig lo up

# Manually configure the eth0 interface. Note that QEMU has a built-in 
# DHCP server that assigns addresses to the hosts starting from 10.0.2.15.
ifconfig eth0 10.0.2.15 netmask 255.255.255.0
route add default gw 10.0.2.1

# Comment this line if changing your keyboard layout is not needed
loadkmap < /etc/host.kmap
EOF

Add executable permission to the rcS script:

$ chmod +x etc/init.d/rcS

Create a compressed filesystem image:

$ find . | cpio -o --format=newc | gzip > ../../rootfs.img.gz

Running and debugging ARM applications

QEMU has all required components prepared (a compressed kernel image and a customized compressed filesystem image) and is ready to run the embedded ARM system.

Run the platform with the following command:

$ cd ~/dev-embedded/ctool-ng
$ ./qemu-0.14.1/arm-softmmu/qemu-system-arm -M versatilepb -m 128M -kernel ./linux-3.2/arch/arm/boot/zImage -initrd ./rootfs.img.gz -append "root=/dev/ram rdinit=/sbin/init" -redir tcp:65000::65000

The redir option will redirect any TCP communication from port 65000 of host system to port 65000 of the guest ARM system. This arbitrarily unused port number is chosen for testing purposes.

The ARM system will boot in a QEMU window then displays the message Please press Enter to activate this console. Press enter and you should be presented with an interactive shell. Start the debugging server as follows:

$ gdbserver --multi localhost:65000

Open a new terminal window, and install the ddd debugger:

$ sudo apt-get install ddd

Edit a configuration file, for example exec-commands.dbg, using the following commands:

$ cd ~/dev-embedded/ctool-ng
$ cat > ./exec-commands.dbg << EOF 
set solib-absolute-prefix nonexistantpath
set solib-search-path $HOME/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/
file ./hello-2.7/src/hello
target extended-remote localhost:65000
set remote exec-file /usr/bin/hello
break main
run
EOF

Run the ddd debugger as follows:

$ export PATH=$PATH:$HOME/x-tools/arm-unknown-linux-gnueabi/bin
$ ddd --debugger arm-unknown-linux-gnueabi-gdb -x exec-commands.dbg

You should see a breakpoint in the main function in hello.c file.

IPsec made simple

Understanding IP security protocol (IPsec) terminology and principles can be a hard task due to the wide range of documentation. This tutorial facilitates this task by providing a succinct documentation and a chronological description of the main steps needed to establish an IPsec tunnel.

Get started

IPsec is a set of protocols developed by the IETF to support secure exchange of packets at the IP layer. IPsec includes protocols for establishing mutual authentication between peers at the beginning of the session and negotiation of cryptographic keys to be used during the session. IPsec is available for both IPv4 and IPv6 and has been deployed widely to implement Virtual Private Networks (VPNs). This tutorial covers IKEv1 and IPv4 only.

Interrelationship of IPsec core documents is illustrated in the following schema:

The Architecture document covers the general concepts, security requirements, definitions, and mechanisms defining IPsec technology.

The ESP Protocol and AH Protocol documents cover the packet format and general issues regarding the respective protocols.

The Encryption Algorithms document is the set of documents describing how various encryption algorithms are used for ESP.

The Authentication Algorithm is the set of documents describing how various authentication algorithms are used for both ESP and AH.

The DOI document contains values needed for the other documents to relate to each other. This includes for example encryption algorithms, authentication algorithms, and operational parameters.

The Key Management Documents are the documents describing the IETF standards-track key management schemes (ISAKMP/Oakley, IKE).

How IPsec works

An IPsec tunnel establishment process can be broken down into five main steps: tunnel initiation, IKE Phase 1, IKE Phase 2, data transfer and tunnel termination.

Tunnel initiation: IPsec tunnel initiation can be triggered manually or automatically when network traffic is flagged for protection according to the IPsec security policy configured in the IPsec peers. In both cases, the Internet Key Exchange Protocol (IKE) process starts.

IKE Phase 1, IKE Phase 2: IKE offers a means to automatically negotiate security parameters and derive suitable keying material. IKE also manages the process of re-creating, or refreshing, frequently keys to ensure data confidentiality between peers. The basic operation of IKE can be broken down into two phases: IKE phase 1 and IKE phase 2.

IKE phase 1: This phase is used to negotiate the parameters and key material required to establish an ISAKMP Security Association (ISAKMP SA). The ISAKMP SA is then used to protect future IKE exchanges and to set up a secure channel for negotiating IPsec SAs in IKE phase 2.

IKE Phase 2: This phase is used to negotiate the parameters and key material required to establish two unidirectional IPsec SAs for incoming and outgoing traffic. The IPSEC SAs are then used to protect network traffic during Data transfer phase.

Data transfer: Incoming and outgoing network traffic is encapsulated according to the bundle of algorithms and parameters provided by their respective negotiated IPsec SA to provide confidentiality and authenticity (ESP protocol) or authenticity only (AH protocol).

Tunnel termination: A tunnel is closed when its IPsec SAs terminate through deletion or by timing out. An IPsec SA can time out when a specified number of seconds have elapsed or when a specified number of bytes have passed through the tunnel.

IKE Phase 1

IKE phase 1’s purpose is to establish a secure authenticated communication channel by using the Diffie–Hellman key exchange algorithm to generate a shared secret key to encrypt further IKE communications. This negotiation results in one single bi-directional ISAKMP SA. The authentication can be performed using either pre-shared key (shared secret), digital signatures, or public key encryption as described by RFC 2409.

IKE Phase 1 operates in either Main Mode or Aggressive Mode. Main Mode protects the identity of the peers but Aggressive Mode does not.

Main Mode has three two-way exchanges between the initiator and the receiver.

    • First exchange: The algorithms and hashes used to secure the IKE communications are agreed upon in matching ISAKMP SAs in each peer.
    • Second exchange: Uses a Diffie-Hellman exchange to generate shared secret keying material used to generate shared secret keys and to pass nonces numbers sent to the other party and then signed and returned to prove their identity.
    • Third exchange: Verifies the other side’s identity (the identity value is an IP address, an FQDN, an email address, a DNS or a KEY ID form in encrypted form). The main outcome of main mode is matching ISAKMP SAs between peers to provide a protected pipe for subsequent protected ISAKMP exchanges between the IKE peers. The ISAKMP SA specifies values for the IKE exchange: the authentication method used, the encryption and hash algorithms, the Diffie-Hellman group used, the lifetime of the ISAKMP SA in seconds or kilobytes, and the shared secret key values for the encryption algorithms. The ISAKMP SA in each peer is bi-directional.

Aggressive Mode has fewer exchanges with fewer packets. On the first exchange, almost everything is squeezed into the proposed ISAKMP SA values: the Diffie-Hellman public key (a nonce that the other party signs) and an identity packet, which can be used to verify identity via a third party. The receiver sends everything back that is needed to complete the exchange. The only thing left is for the initiator to confirm the exchange. The weakness of using the aggressive mode is that both sides have exchanged information before there’s a secure channel. Therefore, it’s possible to sniff the wire and discover who formed the new SA. However, it is faster than main mode.

NAT Traversal also known as UDP encapsulation is a general term for techniques that establish and maintain Internet protocol connections traversing network address translation (NAT) gateways and devices. RFC 3947 defines the negotiation during the Internet key exchange (IKE) phase and RFC 3948 defines the UDP encapsulation.

Dead Peer Detection (DPD) is used to monitor the peer and quickly detect when it gets unreachable. It works by exchanging probe packets, and if the peer does not answer for some time, the security associations are killed. DPD is documented by RFC 3706.

XAUTH, Mode config and Hybrid authentication are optional extensions of the IKE phase 1 described in RFC drafts.

Extended Authentication (XAUTH) provides additional user authentication by prompting the user for a username and password.

Mode config is used to deliver parameters such as IP address and DNS address to the client.

Hybrid authentication makes the IKE phase 1 asymmetric: the VPN IPsec server authenticates to the IPsec clients by using a certificate, and the client does not authenticate in IKE phase 1. This extension is usually used with XAUTH to provide a high security level.

IKE Phase 2

The purpose of IKE phase 2 is to negotiate IPsec SAs to set up the IPsec tunnel. IKE phase 2 performs the following functions:

    • Negotiates IPsec SA parameters protected by an existing ISAKMP SA.
    • Establishes IPsec security associations.
    • Periodically renegotiates IPsec SAs to ensure security.
    • Optionally performs an additional Diffie-Hellman exchange.

IKE phase 2 has one mode called quick mode. Quick mode occurs after IKE has established the secure tunnel in IKE phase 1. It negotiates a shared IPsec policy, derives shared secret keying material used for the IPsec security algorithms, and establishes IPsec SAs. Quick mode exchanges nonces that provide replay protection. The nonces are used to generate new shared secret key material and prevent replay attacks from generating bogus SAs.

Quick mode is also used to renegotiate a new IPsec SA when the IPsec SA lifetime expires. Base quick mode is used to refresh the keying material used to create the shared secret key based on the keying material derived from the Diffie-Hellman exchange in IKE phase 1.

If perfect forward secrecy (PFS) is specified in the IPsec policy, a new Diffie-Hellman exchange is performed with each quick mode, providing keying material that has greater entropy (key material life) and thereby greater resistance to cryptographic attacks. Each Diffie-Hellman exchange requires large exponentiations, thereby increasing CPU use and exacting a performance cost.

When the IPsec SAs terminate, the keys are also discarded. When subsequent IPsec SAs are needed for a flow, IKE performs a new IKE phase 2 and, if necessary, a new IKE phase 1 negotiation. A successful negotiation results in new IPsec SAs and new keys. New IPsec SAs can be established before the existing SAs expire, so that a given flow can continue uninterrupted.

Data transfer

IPsec protocols are AH (Authentication Header) and ESP (Encapsulating Security Payloads):
AH is a format protocol defined in RFC 2402 that provides data authentication, integrity, and non repudiation but does not provide data confidentiality. This protocol has largely been superseded by ESP.
ESP is a format protocol defined in RFC 2406 that provides data confidentiality, integrity and data origin authentication, replay attack protection.
ESP supports the use of symmetric encryption algorithms, including DES, 3DES, and AES, for confidentiality and the use of MD5 HMAC and SHA1 HMAC for data authentication and integrity.

AH and ESP protocols support two modes of use: Transport and Tunnel.
Transport mode encrypts only the data portion of each packet, but leaves the header untouched. Tunnel mode is more secure and encrypts both the header and the payload.

The figures below describe the most common ways to encapsulate original IP packets:

Tunnel/Transport modes using AH protocol

 Tunnel/Transport modes using ESP protocol