development

F2F Computing Development Documentation

This is about the development efforts in the new version of F2F, using a c-core and providing backends to various langauages.

Project organization

F2F has a modular architecture which allows to port it to the platforms with limited set of resources.

The Core Module

The main F2F module responsible for peer management in F2F computing groups, inter-peer communication and running the application code. The core has a set of interfaces accessible to the dependent modules, which are registered to the Core at boot-time. Core needs at least one Computing- and one Communication module to be registered in order to run the distributed applications. In order to register new peers, to create computing groups, to start, control and monitor the running distributed applications the core (and attached computing and communication modules) need to be wrapped by UI-front end. The front-end is responsible for wrapping the Core interfaces with GUI elements, console commands or configuration scripts.

Communication Modules

The distributed computing applications need a fast communication between the running instances, hence the direct inter-peer connections are more preferable then a relayed one. However in case when direct connection is impossible the relayed one can be used. The communication is then relayed over existing peer or a set of peers. There is number of communication protocols (TCP, UDP, SCTP, Bluetooth OBEX ... ) and instant messaging protocols (XMPP, MSN, ICQ ... ), hence the need for multiple communication modules.

Computing Modules

The distributed applications can be written in different languages, the High Performance Computing applications usually require low level languages like C/C++ and Fortran. At the same time prototyping the distributed applications requires higher level languages like Python. The support of mainstream languages (Java,C#) attract more developers to write the distributed applications for Friend-to-Friend platform.

Development Requirements

Development Utilites (SDK, runtime and development dependencies): On Windows: As F2F was originally developed on Linux, the GNU development utilities are required on Windows. We use MinGW and MSysplatforms to emulate GNU utilities on windows. Use automatic installer to install the MinGW on windows. **** On All Platforms To build the F2F computing on various platforms we developed a python based build system which includes Python scripts, SConstructor scripts and SWIG generator interface definitions. In order to run the build scripts the following tools have to be installed:

Python 2.6.6 (python and python-dev packages on Linux, Python 2.6.6 installer on Windows)
SCons >= 2.0.0 (scons package on Linux, SCons installer on Windows)
SWIG >=1.3.0 (swig package on Linux, SWIG archive on Windows)

NB! On Linux all the utilities are by default visible in system path, on Windows you have check if the corresponding utilities (python, scons, swig) are in system path. If these are not there you have to add them manually, otherwise if some of them is missing the build scripts will be not runnable or failing in the middle of build process. The most of the Modules of F2F (Core, LLVM, Communication) are written in pure C or C++. In order to build these Modules the GNU C/C++ compiler has to be installed (build-essential package on Linux, MinGW environment on Windows). **** Some of the F2F modules (like Python Computing Engine, Python Console Front-End and Java Console Front-End) are platform independent and require only the corresponding SDK to be installed in the system.

Java JDK 1.6.0 for Java Based Modules (openjdk-6-jdk and openjdk-6-jre on Linux, Oracle Java jdk1.6.0 on Windows)
Python 2.6.6 (including development headers) for Python Based Modules (python and python-dev packages on Linux, Python 2.6.6 installer on Windows)

On Linux: Because the original development environment for F2F is Linux, some modules are strictly Linux oriented (LLVM Computing Engine, Pidgin-Plugin Front-End). In order to build these packages the following development packages need to be installed on Linux:

pidgin-dev (and all its dependencies) - Required by Pidgin-Plugin Frond-End
llvm-dev (2.6 or 2.7 on Debian 6.0, 2.8 on Ubuntu 10.10)

The support of the other platforms by these modules depends on the development resources, which are currently limited.

Eclipse IDE

The Friend-to-Friend Computing development evolves multiple languages including C/C++, Java and Python. Ofter there is a need to quickly switch between the languages keeping all the highlighting and autocomplete features of the IDE. Eclipse is the one which supports this feature. In order to develop the Friend-to-Friend Computing in Eclipse the following additional plugins should be installed:

eclipse cdt (C/C++ development environment)
eclipse pydev (Python development environment)

In addition you can install egit to integrate git version control system into eclipse, however git original utilities gitk and git-gui are far more advanced.

Git - Fast Version Control System

We keep out code in the most advanced opensource distributed version control system (Git), which was developed by the authors of Linux Kernel.

In windows you just use the installer package from the Git downloads page.
In Linux you should install git-all package from the repository.

If you are new to the Git, please consider the following manual we provided for our students.

Openfire XMPP Server

Friend-to-Friend Communication modules are heavily dependent on Instant Messaging. The instant communication is used to bootstrap the Peer-to-Peer communication, hence the instant messaging modules should be tested well. For offline-testing the XMPP based Instant Messaging we used Openfire XMPP Server. This one is Java based and has nice Web based management interface. You can use this one to run the same offline-tests we do by following next steps:

Install Openfire locally
Disable offline message storage
Create accounts f2f01,f2f02,f2f03 and f2f04 with password f2f
Using Pidgin-Plugin run 4 instances locally (or in your local network) using 4 accounts you just created.
Associate the accounts (all-to-all and select the Initiator or at least all-to-one which will be Initiator)
On the Initiator instance create F2F-Chat and invite remaining Instaces
Submit the Distributed Application for testing
Using Python Console Front-End, follow steps (1-2) form previous list
The testing scripts are held in /adapters/im/headless/src/tests
Modify f2fSlave.py, f2fSlave2.py, f2fSlave3.py, change the server name, login name and password accordingly to the accounts you just created
Modify f2fMaster.py in the same way in addition specify 3-instances you will use in computing group (use the accounts you created)
In master script specify the distributed application you wish to run
Run slave instances using bash script (sh run.sh f2fSlave.py)
Run master instance using bash script (sh run.sh f2fMaster.py)
To terminate instance simply hit any key during runtime

Example applications can be found in the source folders of each computing modules

Quick Development Setup in Linux

In Debian/Ubuntu you just install following packages:

sudo aptitude install git-all build-essential llvm-dev python-dev pidgin-dev openjdk-6-jdk openjdk-6-jre scons swig

If you want to have Eclipse IDE:

sudo aptitude install eclipse eclipse-jdt eclipse-cdt eclipse-pydev

Quick Development Setup in Windows

Joke;) There is no quick way in Windows, you need to manually setup all these tools in addition to MinGW environment.

Getting the code

We assume you followed the previous instructions and installed Git version management system. Our main repository is accessible read-only at http://git.f2f.ulno.net, alternatively through Gitweb interface. To clone the repository use the following Git command:

git clone http://git.f2f.ulno.net f2f-computing

As the F2F project is getting larger and we have more developers evolved, the repository now contains the separate branches for each of them.

The branches:

master - the default branch with F2F stable build, this one is automatically checked by git clone command
headless-java-dev - The Java Console Front-End Development (owned by Vladimir Shkarupelov)
core-dev - The Core Componetnt Development (owned by Artjom Lind)
f2f-dev - The F2F Build Scripts Development, where different F2F modules are put together and tested on different platforms

The tags:

f2f-stable -> indicating the latest stable build of F2F (curently f2f-stable -> stable-llvm-2.8 -> master branch)
f2f-testing -> indicating testing build, where some modules can be detached (currently f2f-testing -> intercfaces)
stable-llvm-2.8 -> stable build LLVM 2.8 + Pidgin-Plugin Front-End + Python Console Front-End, Linux Only
stable-llvm-2.7 -> stable build LLVM 2.7 + Pidgin-Plugin Front-End + Python Console Front-End, Linux Only
stable-llvm-2.6 -> stable build LLVM 2.6 + Pidgin-Plugin Front-End + Python Console Front-End, Linux Only
interfaces ->testing communication, stable Core build on Windows and Linux with new Java and Python interfaces, no Computing Engines
tcp-ip - Initial TCP-IP communication
pidgin-gtk - Pidgin-Plugin Front-End GUI improvements using GTK (previously owned by Madis Kapsi)
old-mobile -> stable build for Mobile Devices (Python Computing Module + Python Console Front-End), Linux, Windows, Symbian S60
f2f-android -> initial Android port of F2F Computing (previously owned by Christophe Colas)
f2f-llvm25 -> initial LLVM 2.5 support (previously owned by Oleg Knut)

Read/Write access:

You need to be a member of the development team in order to have direct write access to the main repository. If you want to join please contact us and send your public ssh-key. If you just want to propose some changes, please send us a link to your published repository or send your changes by mail in git archive.

Building the code

We assume you did set up Python, SCons and SWIG accordingly to our manual. For building navigate to directory where you checked out the Friend-to-Friend Computing

scons -f sconsbuild.py debug=1

For cleaning:

scons -f sconsbuild.py -c

Build Scripts and Project Directory structure

//TODO: Switching between required modules //TODO:Adding new module

Automating Build/Debug in Eclipse

//TODO: Outdated, currently this holds only for F2F Mobile build (old-mobile tag in repository) For automating the build in eclipse, put this line in the External Builder Build command field (select in project - properties - C/C++ Build - Builder settings - External Builder). make sure to unselect Use default build command and Generate Makefiles automatically. As Build directory select ${workspace_loc:/F2FCore/build}. In the Behaviour tab make sure the field next to Build is empty and the field next to Clean contains -c. Now edit in F2FHeadless/src/tests f2fMaster.py and f2fClient.py so that the parameters reflect the servers-settings you use. For testing, run first f2fClient.py and then parallel f2fMaster.py. For convenience and a good debugging experience, create the following four run targets in eclipse:

python
- F2FHeadless master
  - Project: F2FHeadless
  - Main Module: ${workspace_loc:F2FHeadless/src/tests/f2fMaster.py}
  - Arguments: Working directory: Other: ${workspace_loc:F2FHeadless/src/tests}
- F2FHeadless slave
  - Project: F2FHeadless
  - Main Module: ${workspace_loc:F2FHeadless/src/tests/f2fSlave.py}
  - Arguments: Working directory: Other: ${workspace_loc:F2FHeadless/src/tests}
c/c++
- F2Fheadless c-master
  - Project: F2FCore
  - C/C++ Application: /usr/bin/python (or the windows executable)
  - Arguments: Program arguments: f2fMaster.py
  - Arguments: Working directory: ${workspace_loc:F2FHeadless/src/tests}
- F2Fheadless c-slave
  - Project: F2FCore
  - C/C++ Application: /usr/bin/python (or the windows executable)
  - Arguments: Program arguments: f2fSlave.py
  - Arguments: Working directory: ${workspace_loc:F2FHeadless/src/tests}

Then you can debug the f2f in c and in python. Have fun calculating Pi the hard way ;-)