Bohl's Blog

my digital life

building ffmpeg on Windows (4 of n)

This is an update to the previous post - I made the patches work with an x64-build and the tweaked some more files so that FFmpeg now builds fine with these configure-script:

./configure --enable-inline-asm --enable-gpl --enable-nonfree --disable-doc --toolchain=cygicl --host-cc=gcc --arch=x86_32

or

./configure --enable-inline-asm --enable-gpl --enable-nonfree --disable-doc --toolchain=cygicl --host-cc=gcc --arch=x86_64

And of course it passes all FATE-tests.

What you need are these patches

 patches.diff (135.82 kb)

Then fire up a Cygwin-command-prompt with either the x86-ICL or the x64-ICL on the path. In order to use the x64-ICL, just change the batch-file (from the previous post) to

call C:\Progra~2\Intel\Compos~1\bin\ipsxe-comp-vars.bat intel64 vs2012

In order to check which icl you are using, just type 'icl' - if you get this, then it is the x64-version:

The x86-compiler instead gives:

 

Let's start with a fresh check-out of ffmpeg, then you would have to enter something like:

git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

cd ffmpeg

patch -p1 < ../patches.diff

./configure --enable-inline-asm --enable-gpl --enable-nonfree --disable-doc --toolchain=cygicl --host-cc=gcc --arch=x86_64

The above line with "configure" will build the x64-version, for the x86-build you need to enter --arch=x86_32.

Don't forget to have the two scripts iclwrap.sh and tweakpaths.sh in place (e.g. at /usr/local/bin). Just in case, here they are again (I'd think they should be unchanged from the previous post):

tweakpaths.sh (1.30 kb)

iclwrap.sh (1.54 kb)

I tested with ffmpeg as of version 6c4516d0413ea9b2a9b48fb83d0ba0ef7bc84f92.

building ffmpeg on Windows (3 of n)

In this post we will build FFmpeg with the Intel-compiler ICL - with the usual configure-/make-scripts. First of all we need a Cygwin-shell with the Intel-compiler on the path. It is easiest to call the batch-file "ipsxe-comp-vars.bat" which is installed with the Intel-compiler. So we put together a simple batch-file something like this:

@echo off
call C:\Progra~2\Intel\Compos~1\bin\ipsxe-comp-vars.bat ia32 vs2012
chdir /D d:\cygwin\bin
start .\mintty.exe -

Of course you have to substitute your respective paths. Now, when you launch the Cygwin-shell it should look like this:

Let me say beforehand: I tested the following steps with the FFmpeg-sources as of 06/02/2013. The corresponding git-commit-ID is 582f36ca3fb1c69dbe3478f174d36278f5dd3f63. So, if something goes wrong it might because of changes in the FFmpeg-repository. In order to be sure that you get this specific version you need to run this command after downloading the most recent FFmpeg-repository:

git checkout 582f36ca3fb1c69dbe3478f174d36278f5dd3f63

Now, first of all, we need to get the FFmpeg-sources:

git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

Now download the two patches below:

cygicl.diff (31.31 kb) 

inlineassembly.diff (60.75 kb)

The first patch modifies the configure - script, adds some files and applies minimal modifications so that FFmpeg compiles without inline-assembly. The second patch contains more extensive changes to the FFmpeg-code in order to make the inline-assembly work with icl.

I will elaborate on the changes in a latter post, in this post I will just list the steps how to make it compile. Change into the ffmpeg-folder and apply both patches:

patch -p1 < ../cygicl.diff
patch -p1 < ../inlineassembly.diff

Now we still need two more scripts - the first one is a little wrapper for the icl-compiler, and the second is required for executing the FATE-test-suite successfully.

 iclwrap.sh (1.53 kb)

 tweakpaths.sh (1.30 kb)

Place them to somewhere on the (Cygwin-) path - e.g. /usr/local/bin.

We can now run the configure-script:

./configure --enable-inline-asm --disable-doc --toolchain=cygicl --host-cc=gcc

If you chose to not apply the inlineassembly.diff-patch, then you need to specify --disable-inline-asm. This will take a while and should finish without any errors.

Then we are ready to fire up make - which can be speeded up by allowing it to run multiple compiler-processes concurrently. Something like this might be a good choice:

make -j "$(grep -c processor /proc/cpuinfo)"

Again, this should finish without errors.

Finally, we can run FATE with this command:

make  TARGET_EXEC=tweakpaths.sh fate SAMPLES=PATH_TO_FATE_SUITE_SAMPLES  -j "$(grep -c processor /proc/cpuinfo)"

Don't forget to give the correct path (where you have place the fate-samples) instead of PATH_TO_FATE_SUITE_SAMPLES.

At least for me (with the latest icl, version 13.1.2.190 build 20130514) all fate-tests pass.

Adaptec AHA-3940 and Windows7-x64

I happened to have an Adaptec AHA-3940UW-card lying around, and wanted to use it with Windows7-x64 (in order to operate a DLT-tape drive). Yes, I am using good-old tapes for archiving things - I have high hopes that the data on the tapes will last for decades. Now the trouble was - I was unable to find a driver for this card. It seems that only x86-drivers are available. I was about to resign (and either install Windows7-x86 or buy a new card) - when I ran across this thread.

For me this driver works like a charm - I haven't tested it with a hard-drive or so, but with my trusted DLT-tape-drive it works flawlessly.

Adaptec78xx_x64.rar (45.34 kb)

building ffmpeg on Windows (2 of n)

In the last post we built an ffmpeg-binary using the cygwin-gcc-compiler. One of the drawbacks is that it is dependent on a cygwin-environment - it relies on the cygwin-DLLs and can work only inside a complete cygwin-installation. 

The binary cannot deal with traditional Windows-filenames - it will use the translation into "cygwin"-paths. On the one hand, this is an advantage - inside the cygwin-environment everything works smooth.

So, next we will build ffmpeg with the mingw-gcc-compiler. This will give us a binary which will work outside of the cygwin-environment.

The mingw-gcc-compiler is available as a cygwin-package, and we could probably go that way - however, I chose a different approach. We use Zeranoe's mingw-build-script in order to download and build the mingw-gcc-compiler ourselves. This script should do it in one sweep:

#!/bin/bash

ROOTDIR="`pwd`"

getmingwscript() {
cat <<EOF
Now downloading the mingw-installer script from http://www.zeranoe.com.
After the download has completed, the script will be executed.
EOF
zeranoe_url="http://www.zeranoe.com/scripts/mingw_w64_build"
scriptname="mingw-w64-build-3.2.3"
if [ -f $scriptname ];
then
	rm $scriptname
fi
wget $zeranoe_url/$scriptname
mkdir mingw

# for whatever reasons, the svn-checkout sometimes fails (most of the time on a slow machine) -> just run an update in case of failure
sed 's/svn checkout http:\/\/mingw-w64.svn.sourceforge.net\/svnroot\/mingw-w64\/trunk || exit 1/svn checkout http:\/\/mingw-w64.svn.sourceforge.net\/svnroot\/mingw-w64\/trunk || svn update trunk\/ || exit 1/' <$scriptname >mingw/$scriptname 
rm $scriptname

cd mingw
chmod +x ./$scriptname
./$scriptname --build-type=win32 --cpu-count="$(grep -c processor /proc/cpuinfo)" --default-configure
cd ..
}

getffmpeg() {
cd source
git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg
cd ..
}

cleanmingw() {
rm -rf $ROOTDIR/mingw
}

cleansource() {
rm -rf $ROOTDIR/source
}

cleansoftware() {
rm -rf $ROOTDIR/software
}

makesoftwarefolders() {
[ -d $ROOTDIR/software ] || mkdir $ROOTDIR/software
[ -d $ROOTDIR/software/packages ] || mkdir $ROOTDIR/software/packages
[ -d $ROOTDIR/software/packages/win32 ] || mkdir $ROOTDIR/software/packages/win32
[ -d $ROOTDIR/software/packages/win32/lib ] || mkdir $ROOTDIR/software/packages/win32/lib
[ -d $ROOTDIR/software/packages/win32/pkgconfig ] || mkdir $ROOTDIR/software/packages/win32/lib/pkgconfig
[ -d $ROOTDIR/software/packages/win32/include ] || mkdir $ROOTDIR/software/packages/win32/include
}

buildffmpeg() {
cd source/ffmpeg
[ -f config.mak ] && make distclean
export PKG_CONFIG_PATH="$ROOTDIR/software/packages/win32/lib/pkgconfig" 
export LDFLAGS="-L$ROOTDIR/software/packages/win32/lib" 
export CFLAGS="-I$ROOTDIR/software/packages/win32/include"
./configure  --prefix="$ROOTDIR/software/packages/win32" --enable-memalign-hack --arch=x86 --target-os=mingw32 --cross-prefix=i686-w64-mingw32- --enable-gpl --enable-nonfree 
make
make install
}

yes_no_sel () {
unset user_input
question="$1"
shift
while [[ "$user_input" != [YyNn] ]]; do
    echo -n "$question"
    read user_input
    if [[ "$user_input" != [YyNn] ]]; then
        echo "Your selection was not vaild, please try again."; echo ""
    fi
done
}

yes_no_sel "Would you like to start from scratch (download and build mingw)? [y/n]: "
if [[ "$user_input" = [Yy] ]]; then
    cleanmingw
    getmingwscript
fi

cleansource
cleansoftware
[ -d source ] || mkdir source

getffmpeg

export PATH="$PATH:$ROOTDIR/mingw/mingw-w64-i686/bin"

makesoftwarefolders

buildffmpeg

You have to place this script into some folder (and name it e.g. buildffmpeg.sh), then run it. It will ask you whether you want to download and build mingw from scratch - the first time you run it, you will have to say "y[es]", when running it a second time you may choose to skip this step (and use the already existing mingw-gcc-compiler).

After the script has finished its job (without errors I assume...) you should find three new folders: mingw (the sources and the binaries for the mingw-gcc-compiler), source (the ffmpeg-sources) and software (the results from the ffmpeg-build). The ffmpeg-binary is to be found at ...\software\packages\win32\bin.

OK, now the harder part is to run this binary through the FATE-test-suite - the problem is that binary is now expecting Windows-filenames, and FATE (being a Unix-script) is passing Unix-filenames to the executable. We might be able to get away by arranging our directory-layout in a way that Windows- and Unix-syntax come out to the same result, but there is another way: FATE allows to specify a wrapper ('TARGET_EXEC') which is used to run the tests, and we can use this wrapper to convert the paths passed to the executable. This script will do the trick:

#! /bin/sh
 
 function tweakcygwinpath()
 {
	local tweaked
	if [[ $1 == *./tests/data/lavf/lavf.gxf ]];
	then
		echo "./tests/data/lavf/lavf.gxf"
	else
		if [[ $1 == *./tests/data/lavf/lavf.mkv ]];
		then
			echo "./tests/data/lavf/lavf.mkv"
		else
			if [[ $1 == */tests/vsynth1/00.pgm ]];
			then
				echo "./tests/vsynth1/00.pgm"
			else
				tweaked="$(cygpath -w $1)"
				echo "$tweaked"
			fi
		fi
	fi
 }
 
 cmd=$1
 
 declare -a argstweaked
 
 shift 1
 index=0
 for ARG in "$@"
 do
    if [[ $ARG == /cygdrive/* || $ARG == /home/* ]];
	then
		argstweaked[$index]=$(tweakcygwinpath $ARG)
	else
		if [[ $ARG == amovie=*/amrwb/seed-12k65.awb,silencedetect=d=-20dB ]];
		then
			filename=`echo "$ARG" | sed -E 's/amovie=(.*),.*|.*/\1/'`
			if [ ! -d "./temp" ]; then
				mkdir ./temp
			fi
			cp "$filename" ./temp/
			argstweaked[$index]="amovie=./temp/seed-12k65.awb,silencedetect=d=-20dB"
		else
			if [[ $ARG == amovie=*/filter/seq-3341-7_seq-3342-5-24bit.flac,ebur128=metadata=1 ]];
			then
				filename=`echo "$ARG" | sed -E 's/amovie=(.*),.*|.*/\1/'`
				if [ ! -d "./temp" ]; then
					mkdir ./temp
				fi	
				cp "$filename" ./temp/
				argstweaked[$index]="amovie=./temp/seq-3341-7_seq-3342-5-24bit.flac,ebur128=metadata=1"
			else
				argstweaked[$index]=$ARG
			fi
		fi
	fi
	
	((index++))
done

$cmd "${argstweaked[@]}"

So now we can run FATE with this command

make  TARGET_EXEC=./tweakpaths.sh fate SAMPLES=/cygdrive/z/fate-suite/

assuming we have placed the above script in a file "tweakpaths.sh" and the FATE-samples are available at /cygdrive/z/fate-suite.

Some comments on the script: lines 37-53 deal with the FATE-tests "filter-metadata-silencedetect" and "filter-metadata-ebur128". The problem is that a path appears in an argument (for a filtergraph iirc), and the parser cannot handle a Windows-path (this is something to look into...). The desperate workaround is to copy the file, so that it can be accessed using a relative path. TODO: we should delete the copied files afterwards.

building ffmpeg on Windows (1 of n)

Now it is time to set up Cygwin: go to http://cygwin.com/ and run the installer setup.exe. For our purposes, the following packages need to be installed:

  • subversion
  • wget
  • yasm
  • bison
  • flex
  • cvs
  • git
  • gcc-core
  • gcc-g++
  • rsync
  • bc
  • make

You will need to select all these packages by hand (at least I do not know about a way to script this).

Here are some screenshots of what you have to select: cygwin-selections.png (378.92 kb)

 

Now launch a Cygwin-shell, make a new folder (where you will build ffmpeg, say "cygwin"). Then get the Git-repository, and you are ready to configure and make ffmpeg - like so:

cd ~
mkdir cygwin
cd cygwin

git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

cd ffmpeg

./configure --enable-memalign-hack

make

 This should work without any flaw, and after a while you should find a newly-built ffmpeg.exe in the folder. If you happen to have more than one CPU on your machine, you might want to use 'make -j <number_of_parallel_jobs>' in order to speed up the build - where number_of_parallel_jobs should be around the number of cores you have. So, if you have 4 cores, then try 'make -j 4'.

Next, let's test-drive the binary and run the FATE-test-suite. This requires to download some test-data, around ~700MB. You may choose to download it to the ffmpeg-folder right away, or you may keep the samples in a separate folder (and re-use it for other builds). In the first case, with this command the samples will be downloaded to a folder "fate-suite" which is created in the ffmpeg-folder

make fate-rsync SAMPLES=fate-suite/

Or if you prefer to download it manually, try something like

rsync -vrltLW --timeout=60 --contimeout=60 rsync://fate-suite.ffmpeg.org/fate-suite/ fate-suite/

Now it is time to run the tests:

make fate  SAMPLES=fate-suite/

You need to tell the path where the FATE-samples are to found as an argument to the SAMPLES-variable.

All tests should work ok, and now have done it: you built the ffmpeg-binary yourself! Of course, this is just the start...

building ffmpeg on Windows (0 of n)

I will try to collect some step-by-step instructions how to build ffmpeg on a Windows-machine. I still find it tedious, and https://winffmpeg.codeplex.com/ is of course not really a viable solution. Above all, it is cumbersome to keep it up-to-date.

OK, my plot is:

  • Describe the setup of Cygwin in order to build ffmpeg with gcc
  • Make the ffmpeg-build work with icl

Stay tuned...

Moving to Windows8 continued...

Some news on my move to Windows8: First of all, now that my Microsoft Surface has arrived, I spent most time with this device. But I don't want to talk about Microsoft Surface here, but on some problems with Windows8 on a good old desktop PC.

I installed Outlook 2010, it sync'ed perfectly well with Outlook.com - the only problem was that S/MIME-support is more or less broken (see e. g. here). I haven't found any usable workaround so far for this curious issue. So, I was happy when Outlook 2013 came out.

All seemed fine, and the new Outlook has built-in support for Outlook.com (and is now using EAS-protocol, no need for the Hotmail-Connector any more). Setting things up is easy (see e. g. here), however - it didn't work for me. Outlook 2013 did not sync and download the emails - for whatever reasons...

How to hard-reset the Packard Bell Liberty Tab G100 tablet

A while back I bought myself an Android tablet (Liberty Tab G100 from Packard Bell). Nothing fancy, just in order to surf the web on the couch and get some hands-on experience with Android. The device was not rooted, no custom ROM on it or so. Well, I am still really impressed by the value one gets for the money, but this post is not about the pros and cons of Android. Suffice to say (from my superficial end-user experience) that the software-platform has its rough edges and glitches.

Now, seemingly for no apparent reason, the device could not be turned on any more. It just showed the (animated) Android-logo for as long as the power-supply would last and that was it. Just this logo and nothing else. So, I read through the forums at xda-developers.com, learned at lot about the various bootloaders, custom roms, adb and what not. However - nothing worked for me. Booting into "recovery mode" by pressing Volume-Down and Power-On didn't help (now I got an Android-robot with an exclamation mark), and ADB I did not get to work. To cut a long story short, what finally made my day is a hint a found here. I am not sure what happened, but the device is workin again. So, here is how to "hard-reset" the device (all data is lost by this operation!):

  1. Power the device off
  2. switch the screen-lock button so that the red is not visible
  3. hold the Volume-Up button and press the power-button
  4. now, when the device starts to vibrate, release power-button and switch the screen-lock several times

Now, you should see some lines of text appear on the screen (just for a second or so in my case), and then the device restarts. If at first you don't succeed with this procedure, try again Foot in mouth. It took me several attempts, I am not sure how many times you have to toogle the screen-lock switch (or when exactly).

I am still curious what was causing the trouble in the first place. The only idea I have so far: the internal memory was completely full (I copied some movies erroneously into the internal memory instead of the SD-card). Should this have caused Android to completely mess up? Well, looking back to the times when computers had a keyboard attached to it, when they had a BIOS or something like this and you could boot them from a floppy disk...

driving ffmpeg

Ffmpeg is one of the open-source projects that have been fascinating me since its first days. I admire the project for what it has archieved, the quality of this piece of code is just marvelous. I had been in the video-editing business for quite some time, and maybe have a good grasp of what wonderful a tool those guys have put together. In particular, the approach to have all pieces in one binary has always attracted me - after having wasted countless hours in DLL- and codec-hell.

However, so far I did not have a chance to put ffmpeg to work in a serious (i.e. paid for) project. Now this chance seems to have arrived, and I am very excited of it. In short, the goal is to use ffmpeg to create movies from a series of images, no audio at this time. So far the implementation uses DirectShow (yes, I am a die-hard DirectShow-fan) - however, DirectShow right out of the box does not offer a lot of video formats. Sure, there are tons of codecs for every format one can think of - but going this route would inevitably mean to enter codec-hell once more. Then, there is MediaFoundation - imho not really better suited for the task at hand, because at least as complex as DirectShow and (again, right out of the box) does not deliver that many new codecs or containers. So, I decided to give ffmpeg a shot.

The first decision is about how to interface with ffmpeg. Three possible choices seem feasible:

  • First create an AVI with DirectShow, then let ffmpeg convert it.
  • Use the ffmpeg-API (i.e. link with libavcodec/libformat and use their API)
  • transfer the source images into ffmpeg, where ffmpeg runs as a stand-alone process, let it encode and output it into a file

The first approach is for sure the lamest - but the easiest. In fact, performance is not that much a concern, so it is not immediately ruled out. The second is probably the most efficient and most solid approach, but there are quite a few drawbacks: besides (potential) legal issues it is about the problem of integrating it into a build-environment, worries about the stability of the API (and problems with updates of ffmpeg) and of course, the complexity of the API itself and the inevitable learning curve (and some more worries). So, I decided to take the third approach.

I was hoping that someone else already tried this out - and was hoping to find a nice library or code snippets for this task. To my astonishment a web search did not bring up many hits, at least not what I was hoping for. And the documentation on the ffmpeg-site itself for this was not too enlighting as well.

The basic idea is to use a named pipe in order to transport the images over to ffmpeg. I got this part (basically) working after some meandering. More on this in one of the next posts...