Bohl's Blog

my digital life

building FFmpeg on Windows (6 of n)

Now that the configure/make-build of FFmpeg is working satisfactorily, it is time to turn to VisualStudio. We will create a VisualStudio-project from the results of the configure/make-build.

The approach I took is somewhat like: we run a configure/make-build which will create the config.h & config.asm files, a makefile and the dependencies. Then we use that information in order to create a VisualStudio-project-file (vcxproj) with the help of a little tool.

After configure has run, we run make with an option telling it to just print the commands it would normally execute:

make --dry-run > makelog.txt

The file "makelog.txt" now contains all compiler invocations, which we just need to parse and turn into a VisualStudio-project. Here is a little tool I put together for this job. I'll need to give it some more polish before releasing the source, so for now just the binary:

MakeVSProject.zip (23.59 kb)

Run this program with the following commandline:

MakeVSProject.exe --projname=vsFFmpeg -m makelog.txt

This will create two new files vsFFmpeg.vcxproj and vsFFmpeg.vcxproj.filters. Open the project with VisualStudio and it should look like this: 

Now, we are almost there, just some more tweaks are required. First of all, the VisualStudio-project includes both x86- and x64-platform, but configure has created config.h & config.asm tailored for just one platform. So we need to add some #ifdef's to these files - that's what the commands below will do:

sed 's/%define ARCH_X86_32 [0-1]//' < config.asm | sed 's/%define ARCH_X86_64 [0-1]//' > config.asm_out
sed 's/%define HAVE_ALIGNED_STACK [0-1]/%if ARCH_X86_32 == 1\n %define HAVE_ALIGNED_STACK 0\n%elif ARCH_X86_64 == 1\n %define HAVE_ALIGNED_STACK 1\n%endif/' < config.asm_out > config.asm
rm config.asm_out

sed 's/#define ARCH_X86_32 [0-1]/#if defined( _M_X64)\n #define ARCH_X86_32 0\n#elif defined( _M_IX86)\n #define ARCH_X86_32 1\n#endif\n/' < config.h | sed 's/#define ARCH_X86_64 [0-1]/#if defined( _M_X64)\n #define ARCH_X86_64 1\n#elif defined( _M_IX86)\n #define ARCH_X86_64 0\n#endif\n/' > config.h_out
sed 's/#define HAVE_ALIGNED_STACK [0-1]/#if defined( _M_X64)\n #define HAVE_ALIGNED_STACK 1\n#elif defined( _M_IX86)\n #define HAVE_ALIGNED_STACK 0\n#endif\n/' < config.h_out > config.h
rm config.h_out

Now the x86- and x64-Release builds should work right out of the box. There are still problems with the x64-Debug and x86-Debug build. With the former one you will get

 

and with the latter something like

 

I will have some more patches for this in the next post.

Of course, the VisualStudio-project still leaves some things to be desired. Foremost, it only includes the files which are producing an object file, headers and files that are only included are missing. I'd think it should be easy to parse the dependency-files (".d") in order to get them. Watch out for an update - again in a later post.

building ffmpeg on Windows (4 of n)

This is an update to the previous post - I made the patches work with an x64-build and the tweaked some more files so that FFmpeg now builds fine with these configure-script:

./configure --enable-inline-asm --enable-gpl --enable-nonfree --disable-doc --toolchain=cygicl --host-cc=gcc --arch=x86_32

or

./configure --enable-inline-asm --enable-gpl --enable-nonfree --disable-doc --toolchain=cygicl --host-cc=gcc --arch=x86_64

And of course it passes all FATE-tests.

What you need are these patches

 patches.diff (135.82 kb)

Then fire up a Cygwin-command-prompt with either the x86-ICL or the x64-ICL on the path. In order to use the x64-ICL, just change the batch-file (from the previous post) to

call C:\Progra~2\Intel\Compos~1\bin\ipsxe-comp-vars.bat intel64 vs2012

In order to check which icl you are using, just type 'icl' - if you get this, then it is the x64-version:

The x86-compiler instead gives:

 

Let's start with a fresh check-out of ffmpeg, then you would have to enter something like:

git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

cd ffmpeg

patch -p1 < ../patches.diff

./configure --enable-inline-asm --enable-gpl --enable-nonfree --disable-doc --toolchain=cygicl --host-cc=gcc --arch=x86_64

The above line with "configure" will build the x64-version, for the x86-build you need to enter --arch=x86_32.

Don't forget to have the two scripts iclwrap.sh and tweakpaths.sh in place (e.g. at /usr/local/bin). Just in case, here they are again (I'd think they should be unchanged from the previous post):

tweakpaths.sh (1.30 kb)

iclwrap.sh (1.54 kb)

I tested with ffmpeg as of version 6c4516d0413ea9b2a9b48fb83d0ba0ef7bc84f92.

building ffmpeg on Windows (3 of n)

In this post we will build FFmpeg with the Intel-compiler ICL - with the usual configure-/make-scripts. First of all we need a Cygwin-shell with the Intel-compiler on the path. It is easiest to call the batch-file "ipsxe-comp-vars.bat" which is installed with the Intel-compiler. So we put together a simple batch-file something like this:

@echo off
call C:\Progra~2\Intel\Compos~1\bin\ipsxe-comp-vars.bat ia32 vs2012
chdir /D d:\cygwin\bin
start .\mintty.exe -

Of course you have to substitute your respective paths. Now, when you launch the Cygwin-shell it should look like this:

Let me say beforehand: I tested the following steps with the FFmpeg-sources as of 06/02/2013. The corresponding git-commit-ID is 582f36ca3fb1c69dbe3478f174d36278f5dd3f63. So, if something goes wrong it might because of changes in the FFmpeg-repository. In order to be sure that you get this specific version you need to run this command after downloading the most recent FFmpeg-repository:

git checkout 582f36ca3fb1c69dbe3478f174d36278f5dd3f63

Now, first of all, we need to get the FFmpeg-sources:

git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

Now download the two patches below:

cygicl.diff (31.31 kb) 

inlineassembly.diff (60.75 kb)

The first patch modifies the configure - script, adds some files and applies minimal modifications so that FFmpeg compiles without inline-assembly. The second patch contains more extensive changes to the FFmpeg-code in order to make the inline-assembly work with icl.

I will elaborate on the changes in a latter post, in this post I will just list the steps how to make it compile. Change into the ffmpeg-folder and apply both patches:

patch -p1 < ../cygicl.diff
patch -p1 < ../inlineassembly.diff

Now we still need two more scripts - the first one is a little wrapper for the icl-compiler, and the second is required for executing the FATE-test-suite successfully.

 iclwrap.sh (1.53 kb)

 tweakpaths.sh (1.30 kb)

Place them to somewhere on the (Cygwin-) path - e.g. /usr/local/bin.

We can now run the configure-script:

./configure --enable-inline-asm --disable-doc --toolchain=cygicl --host-cc=gcc

If you chose to not apply the inlineassembly.diff-patch, then you need to specify --disable-inline-asm. This will take a while and should finish without any errors.

Then we are ready to fire up make - which can be speeded up by allowing it to run multiple compiler-processes concurrently. Something like this might be a good choice:

make -j "$(grep -c processor /proc/cpuinfo)"

Again, this should finish without errors.

Finally, we can run FATE with this command:

make  TARGET_EXEC=tweakpaths.sh fate SAMPLES=PATH_TO_FATE_SUITE_SAMPLES  -j "$(grep -c processor /proc/cpuinfo)"

Don't forget to give the correct path (where you have place the fate-samples) instead of PATH_TO_FATE_SUITE_SAMPLES.

At least for me (with the latest icl, version 13.1.2.190 build 20130514) all fate-tests pass.

Adaptec AHA-3940 and Windows7-x64

I happened to have an Adaptec AHA-3940UW-card lying around, and wanted to use it with Windows7-x64 (in order to operate a DLT-tape drive). Yes, I am using good-old tapes for archiving things - I have high hopes that the data on the tapes will last for decades. Now the trouble was - I was unable to find a driver for this card. It seems that only x86-drivers are available. I was about to resign (and either install Windows7-x86 or buy a new card) - when I ran across this thread.

For me this driver works like a charm - I haven't tested it with a hard-drive or so, but with my trusted DLT-tape-drive it works flawlessly.

Adaptec78xx_x64.rar (45.34 kb)

building ffmpeg on Windows (2 of n)

In the last post we built an ffmpeg-binary using the cygwin-gcc-compiler. One of the drawbacks is that it is dependent on a cygwin-environment - it relies on the cygwin-DLLs and can work only inside a complete cygwin-installation. 

The binary cannot deal with traditional Windows-filenames - it will use the translation into "cygwin"-paths. On the one hand, this is an advantage - inside the cygwin-environment everything works smooth.

So, next we will build ffmpeg with the mingw-gcc-compiler. This will give us a binary which will work outside of the cygwin-environment.

The mingw-gcc-compiler is available as a cygwin-package, and we could probably go that way - however, I chose a different approach. We use Zeranoe's mingw-build-script in order to download and build the mingw-gcc-compiler ourselves. This script should do it in one sweep:

#!/bin/bash

ROOTDIR="`pwd`"

getmingwscript() {
cat <<EOF
Now downloading the mingw-installer script from http://www.zeranoe.com.
After the download has completed, the script will be executed.
EOF
zeranoe_url="http://www.zeranoe.com/scripts/mingw_w64_build"
scriptname="mingw-w64-build-3.2.3"
if [ -f $scriptname ];
then
	rm $scriptname
fi
wget $zeranoe_url/$scriptname
mkdir mingw

# for whatever reasons, the svn-checkout sometimes fails (most of the time on a slow machine) -> just run an update in case of failure
sed 's/svn checkout http:\/\/mingw-w64.svn.sourceforge.net\/svnroot\/mingw-w64\/trunk || exit 1/svn checkout http:\/\/mingw-w64.svn.sourceforge.net\/svnroot\/mingw-w64\/trunk || svn update trunk\/ || exit 1/' <$scriptname >mingw/$scriptname 
rm $scriptname

cd mingw
chmod +x ./$scriptname
./$scriptname --build-type=win32 --cpu-count="$(grep -c processor /proc/cpuinfo)" --default-configure
cd ..
}

getffmpeg() {
cd source
git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg
cd ..
}

cleanmingw() {
rm -rf $ROOTDIR/mingw
}

cleansource() {
rm -rf $ROOTDIR/source
}

cleansoftware() {
rm -rf $ROOTDIR/software
}

makesoftwarefolders() {
[ -d $ROOTDIR/software ] || mkdir $ROOTDIR/software
[ -d $ROOTDIR/software/packages ] || mkdir $ROOTDIR/software/packages
[ -d $ROOTDIR/software/packages/win32 ] || mkdir $ROOTDIR/software/packages/win32
[ -d $ROOTDIR/software/packages/win32/lib ] || mkdir $ROOTDIR/software/packages/win32/lib
[ -d $ROOTDIR/software/packages/win32/pkgconfig ] || mkdir $ROOTDIR/software/packages/win32/lib/pkgconfig
[ -d $ROOTDIR/software/packages/win32/include ] || mkdir $ROOTDIR/software/packages/win32/include
}

buildffmpeg() {
cd source/ffmpeg
[ -f config.mak ] && make distclean
export PKG_CONFIG_PATH="$ROOTDIR/software/packages/win32/lib/pkgconfig" 
export LDFLAGS="-L$ROOTDIR/software/packages/win32/lib" 
export CFLAGS="-I$ROOTDIR/software/packages/win32/include"
./configure  --prefix="$ROOTDIR/software/packages/win32" --enable-memalign-hack --arch=x86 --target-os=mingw32 --cross-prefix=i686-w64-mingw32- --enable-gpl --enable-nonfree 
make
make install
}

yes_no_sel () {
unset user_input
question="$1"
shift
while [[ "$user_input" != [YyNn] ]]; do
    echo -n "$question"
    read user_input
    if [[ "$user_input" != [YyNn] ]]; then
        echo "Your selection was not vaild, please try again."; echo ""
    fi
done
}

yes_no_sel "Would you like to start from scratch (download and build mingw)? [y/n]: "
if [[ "$user_input" = [Yy] ]]; then
    cleanmingw
    getmingwscript
fi

cleansource
cleansoftware
[ -d source ] || mkdir source

getffmpeg

export PATH="$PATH:$ROOTDIR/mingw/mingw-w64-i686/bin"

makesoftwarefolders

buildffmpeg

You have to place this script into some folder (and name it e.g. buildffmpeg.sh), then run it. It will ask you whether you want to download and build mingw from scratch - the first time you run it, you will have to say "y[es]", when running it a second time you may choose to skip this step (and use the already existing mingw-gcc-compiler).

After the script has finished its job (without errors I assume...) you should find three new folders: mingw (the sources and the binaries for the mingw-gcc-compiler), source (the ffmpeg-sources) and software (the results from the ffmpeg-build). The ffmpeg-binary is to be found at ...\software\packages\win32\bin.

OK, now the harder part is to run this binary through the FATE-test-suite - the problem is that binary is now expecting Windows-filenames, and FATE (being a Unix-script) is passing Unix-filenames to the executable. We might be able to get away by arranging our directory-layout in a way that Windows- and Unix-syntax come out to the same result, but there is another way: FATE allows to specify a wrapper ('TARGET_EXEC') which is used to run the tests, and we can use this wrapper to convert the paths passed to the executable. This script will do the trick:

#! /bin/sh
 
 function tweakcygwinpath()
 {
	local tweaked
	if [[ $1 == *./tests/data/lavf/lavf.gxf ]];
	then
		echo "./tests/data/lavf/lavf.gxf"
	else
		if [[ $1 == *./tests/data/lavf/lavf.mkv ]];
		then
			echo "./tests/data/lavf/lavf.mkv"
		else
			if [[ $1 == */tests/vsynth1/00.pgm ]];
			then
				echo "./tests/vsynth1/00.pgm"
			else
				tweaked="$(cygpath -w $1)"
				echo "$tweaked"
			fi
		fi
	fi
 }
 
 cmd=$1
 
 declare -a argstweaked
 
 shift 1
 index=0
 for ARG in "$@"
 do
    if [[ $ARG == /cygdrive/* || $ARG == /home/* ]];
	then
		argstweaked[$index]=$(tweakcygwinpath $ARG)
	else
		if [[ $ARG == amovie=*/amrwb/seed-12k65.awb,silencedetect=d=-20dB ]];
		then
			filename=`echo "$ARG" | sed -E 's/amovie=(.*),.*|.*/\1/'`
			if [ ! -d "./temp" ]; then
				mkdir ./temp
			fi
			cp "$filename" ./temp/
			argstweaked[$index]="amovie=./temp/seed-12k65.awb,silencedetect=d=-20dB"
		else
			if [[ $ARG == amovie=*/filter/seq-3341-7_seq-3342-5-24bit.flac,ebur128=metadata=1 ]];
			then
				filename=`echo "$ARG" | sed -E 's/amovie=(.*),.*|.*/\1/'`
				if [ ! -d "./temp" ]; then
					mkdir ./temp
				fi	
				cp "$filename" ./temp/
				argstweaked[$index]="amovie=./temp/seq-3341-7_seq-3342-5-24bit.flac,ebur128=metadata=1"
			else
				argstweaked[$index]=$ARG
			fi
		fi
	fi
	
	((index++))
done

$cmd "${argstweaked[@]}"

So now we can run FATE with this command

make  TARGET_EXEC=./tweakpaths.sh fate SAMPLES=/cygdrive/z/fate-suite/

assuming we have placed the above script in a file "tweakpaths.sh" and the FATE-samples are available at /cygdrive/z/fate-suite.

Some comments on the script: lines 37-53 deal with the FATE-tests "filter-metadata-silencedetect" and "filter-metadata-ebur128". The problem is that a path appears in an argument (for a filtergraph iirc), and the parser cannot handle a Windows-path (this is something to look into...). The desperate workaround is to copy the file, so that it can be accessed using a relative path. TODO: we should delete the copied files afterwards.

building ffmpeg on Windows (1 of n)

Now it is time to set up Cygwin: go to http://cygwin.com/ and run the installer setup.exe. For our purposes, the following packages need to be installed:

  • subversion
  • wget
  • yasm
  • bison
  • flex
  • cvs
  • git
  • gcc-core
  • gcc-g++
  • rsync
  • bc
  • make

You will need to select all these packages by hand (at least I do not know about a way to script this).

Here are some screenshots of what you have to select: cygwin-selections.png (378.92 kb)

 

Now launch a Cygwin-shell, make a new folder (where you will build ffmpeg, say "cygwin"). Then get the Git-repository, and you are ready to configure and make ffmpeg - like so:

cd ~
mkdir cygwin
cd cygwin

git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

cd ffmpeg

./configure --enable-memalign-hack

make

 This should work without any flaw, and after a while you should find a newly-built ffmpeg.exe in the folder. If you happen to have more than one CPU on your machine, you might want to use 'make -j <number_of_parallel_jobs>' in order to speed up the build - where number_of_parallel_jobs should be around the number of cores you have. So, if you have 4 cores, then try 'make -j 4'.

Next, let's test-drive the binary and run the FATE-test-suite. This requires to download some test-data, around ~700MB. You may choose to download it to the ffmpeg-folder right away, or you may keep the samples in a separate folder (and re-use it for other builds). In the first case, with this command the samples will be downloaded to a folder "fate-suite" which is created in the ffmpeg-folder

make fate-rsync SAMPLES=fate-suite/

Or if you prefer to download it manually, try something like

rsync -vrltLW --timeout=60 --contimeout=60 rsync://fate-suite.ffmpeg.org/fate-suite/ fate-suite/

Now it is time to run the tests:

make fate  SAMPLES=fate-suite/

You need to tell the path where the FATE-samples are to found as an argument to the SAMPLES-variable.

All tests should work ok, and now have done it: you built the ffmpeg-binary yourself! Of course, this is just the start...

building ffmpeg on Windows (0 of n)

I will try to collect some step-by-step instructions how to build ffmpeg on a Windows-machine. I still find it tedious, and https://winffmpeg.codeplex.com/ is of course not really a viable solution. Above all, it is cumbersome to keep it up-to-date.

OK, my plot is:

  • Describe the setup of Cygwin in order to build ffmpeg with gcc
  • Make the ffmpeg-build work with icl

Stay tuned...

Moving to Windows8 continued...

Some news on my move to Windows8: First of all, now that my Microsoft Surface has arrived, I spent most time with this device. But I don't want to talk about Microsoft Surface here, but on some problems with Windows8 on a good old desktop PC.

I installed Outlook 2010, it sync'ed perfectly well with Outlook.com - the only problem was that S/MIME-support is more or less broken (see e. g. here). I haven't found any usable workaround so far for this curious issue. So, I was happy when Outlook 2013 came out.

All seemed fine, and the new Outlook has built-in support for Outlook.com (and is now using EAS-protocol, no need for the Hotmail-Connector any more). Setting things up is easy (see e. g. here), however - it didn't work for me. Outlook 2013 did not sync and download the emails - for whatever reasons...