Over the years, the use of a mobile phone has developed and no longer reflects a device that simply calls and sends text messages. Many mobile devices on the market are actually called smart phones since they offer more advanced computing ability and connectivity than a basic phone. They allow the user to install and run more advanced applications based on a specific platform.
A recent study by ComCore in 2010 shows that 45.5 million people own a smartphone in the United States alone. This clearly shows the huge fast growing segment of the mobile phone market and therefore the importance in mobile software development.
Mobile software development relies heavily on the Operating System foundation found on the mobile device. The Symbian OS was designed from the start for mobile devices and was made popular by the success of Nokia. This helped gain huge market share of more than 50% of other OS's. This made way for 3rd party apps written for the Symbian device and was seen as a start in mobile apps popularity. Mobile application development was taken to the next level with the introduction of Apple's iPhone and the beginning of Apple's "App Store". The unexpected huge success of one of the most profitable mobile software market place made way for major companies like Google's open source Android to invest and give importance to mobile application development.
Objectives
This Group APT Project consists of designing and implementing a mobile application written with Python high-level language within a Symbian (S60) environment on a Nokia device.
The function of this mobile application is to open a TCP/IP connection between a server and the mobile. From a desktop PC, a user chooses various requests to access the mobile device remotely.
These requests can be:
Capturing an image
Capturing a video clip with audio
Open a live streaming connection from mobile device to be outputted and viewed on APT website
Both capturing image and video commands save the multimedia file on the device and on the web server. The latter makes it possible for anyone to view the captured contents on the published APT website.
The entire system is explained in further detail in the next sections.
System Design Considerations
Image and Video Capturing
Diagram
Abstract
The server is used as a web hosting server and also as a link between the mobile placed in a remote location within the same network and the client who sends a request from our website.
Python compiler is installed on the mobile and continuously runs a program that waits and accepts a TCP/IP connection from the client and executes according to the request. Once the image or video has been captured, Python is instructed to upload the multimedia file onto the server
On the other hand, the client program runs on a web server which was written in PHP. Basically, it opens a TCP/IP connection with the mobile's TCP/IP server and sends a command packet on this link asking for the mobile to perform a multimedia task. The web server then waits for the received packet.
Finally this file as well as every other similar files are readily available on the website for the client to view.
Live Streaming
Diagram
Abstract
Live streaming from a mobile device was done by connecting an iPhone mobile with the server using "PocketCam" software so as to enable the mobile's camera to be viewed on the server as an external webcam. The video stream is then streamed locally on the server using HTTP protocol and transcoded as a .ogg file. This stream is then embedded on the website using HTML5 <video> code. Since the html file is hosted on the server, the stream can be viewed from other PCs connected on the same network of the server.
Resources
2 smartphones
Home PC running the server [IIS Services, PHP, Quicktime, VLC]
Router for private network
Personal laptop to serve as client of website
Python program [compiller]
"PocketCam" mobile and PC software
Server Side
A server is a computer designed to process requests and send data to clients all over the network servers are usually under an intensive load from various clients and so they are equipped with extra memory and storage. They also contain programs which can make the server do multiple tasks. Some types of servers are Application server; Audio/Video Server which provides multimedia to websites so media could be easily broadcasted, chat server which provides real time discussion capabilities; FTP (File Transfer Protocol) server provides an ideal service for file sharing and at the same time having a secure connection; Mail server is a server which provides an emailing server, this one is commonly used everyday; Web Server is designed to host a website through HTTP which can be accessed with any browser.
First thing we had to determine what the ideal operating system to host the server is. After looking at the various options we had available we chose Windows 2008 server 64bit version because of security reasons we installed the server on a virtual machine and the host operating was also 64bit.
A 64 bit version was used because this gave us the ability to be able to access more than 4gb of ram on the computer, by this we mean better performance. Programs written can take advantage of the 64 bit architecture and enhance their execution time.
A virtual machine means that a computer is implemented on a host machine in a software state. The downside from this that the hardware is accessed indirectly and therefore causing a small loss in performance.
A remote desktop facility was set up so as to give us an easy way to access the server when we don't have physical access to it, this helped us loads to work as a group.
We used the IIS services to provide the base for the website. First we assigned the website with the option of having various imp addresses as our imp address is not static. We used the default directory for storing the website which is the wwwroot folder. Using the services of a DynDNS we assigned a domain name to our website. As we knew that videos files where going to be played on this web site the MIMEs were configured so as the server would also support 3gp and mp4 formats.
We then installed the PHP on the server so as we would be able to run php script through our website. Several versions were tried because of compatibility issues and missing libraries which without them would render our code useless.
The server firewall settings were also modified because of the high security a server operating systems offers.
When the server was setup Port Forwarding on the router was configured so the pathways for data would be opened. Applications communicate through the internet using different ports so like that data can be divided for different services, for example port 80 is used for HTTP service and port 3389 for a Remote Desktop Connection.
After all this basic preparation the html and PHP files were introduced to the web server, after some modification of the code because of some small mistakes the site was up and running ready to start testing to mobile application.
Website
Using HTML for the layout of our website, the structure is based upon the exploitation of frames. HTML is a HyperText Markup Language which is mostly used to build websites. By inserting the HTML tags <html></html>, one is left to an open canvas which is ready to expose creativity and innovation. After obtaining a DNS name, the files related to the website were added to the 'wwwroot' folder found in the 'inetpub'. Each page in our website, including the homepage, consists of three rows:
Row 1 - inserts the title image
Row 2 - the only area providing user interface
Row 3 - simply states that the website was solely established for a project for the University of Malta
HTML was used for linking images, inserting plug-ins and video players, and maneuvering from one page to another. On the other hand, when sending a request (or rather sending a packet consisting of a string of data) to a mobile placed at a remote location, PHP provided more useful functions for establishing a TCP/IP connection with the mobile and transferring requests to carry out one of the following tasks:
Take a photo and allow it to be viewed on the website
Capture a 10-second video and allow it to be viewed on the website
Last but not least, HTML5 was utilized for live streaming for the mobile's camera. As HTML constantly evolves, recent development has lead to the birth of HTML5. The most important feature of this latter language is its ability to eliminate the need for plug-ins which in fact was a problem that we were challenged with as can be seen in the section of 'Methods of Testing'. In other words, HTML5 cuts out the middle man and simply stream via a '.ogg' file which is designed purposely to enhance streaming efficiency and quality.
Image and Video Capturing
This Multimedia mobile application is a client-server architecture based system, the information flow is not standalone but it goes through a communication network. This comprises a program that communicates with another program across a computer network. The process initiating the communication is called a client process and the program waiting for the communication to be initiated is the server process. Therefore this system incorporates a networked environment allowing a client to send requests to the mobile server and the same server receive these requests and respond to them.
PHP
The client program runs on the web server and the scripts are PHP based. PHP comes with very capable programming functions. In this system the respective PHP scripts includes fundamental socket functions to create a socket-based client server communication over TCP/IP and rapidly build a client-server application in PHP. Basically PHP is a scripting language and is commonly used to dynamically generate web pages. As a result of this the PHP code is embedded into HTML document and interpreted by the web server which generates the web page document. Therefore for this multimedia mobile application PHP was fundamental and effective to create and manipulate socket communication in order to obtain a network-based client-server application.
The communication between the client and server operates over and make use of a connection-oriented protocol such as the TCP/IP which is a basic communication language. TCP/IP is a two layer program, the higher layer is the Transmission control protocol and the lower layer is the Internet protocol. The higher layer manages the assembling of the command packet into smaller packets that are transmitted and received by a TCP layer that reassembles the packets into the original message. The lower layer handles the address part of each packet in order to get to the right destination. TCP/IP is capable of setting up a connection between a client and a server and providing reliable data transmission within that connection. The client-server model of communication which the TCP/IP uses is a computer user (a client) which requests and sends a command to the mobile (a server) in the network. TCP/IP allows end to end security over data connection from the mobile terminal to the network as well as providing fast connections, bringing efficient and interesting multimedia applications such as image and video clips.
Configuration of the communication between the client and server that implements TCP/IP
Configuration of the Image and Video upload
Result Analysis
Capturing an image
The client opens a TCP/IP connection with the mobile TCP/IP server and sends a command on this link asking for the mobile (E71 or 5800) to capture an image.
Python program waiting for a TCP/IP connection.
The mobile accepts the TCP/IP connection from the client; interpret the command and captures the image. The client then reads the incoming image data handed over from Python mobile and uploads the image on the website.
Capturing a video
The client opens a TCP/IP connection with the mobile TCP/IP server and sends a command on this link asking for the mobile (E71 or 5800) to capture a video.
Python program waiting for a TCP/IP connection.
Once the 10 seconds pass, the mobile stops recording and sends the video back to the client. The client then reads the incoming video data handed over from Python mobile and uploads the video on the website.
The mobile accepts the TCP/IP connection from the client; interpret the command and starts capturing the video for 10 seconds.
Live Streaming
The last thing that was done in our APT was trying to achieve video live streaming from a mobile device onto a web server and ultimately be streamed on our APT website.
Many scenarios were initially considered such as the following:
One idea was to code a program with python to take successive pictures with a small time interval in between. These pics will then be uploaded onto the web server using the same image capturing code. Each new picture taken will replace the previous uploaded picture. Thus, on the web server, the image file will be constantly updated to have the latest captured scene. The website where the image is hosted would have to constantly refresh itself to display new images each time. Although this concept seemed a viable solution, we had problems in getting the website to refresh itself and also for the images to constantly upload. Thus it was abandoned.
Another scenario that we tested was using open source VLC software by VideoLan. The VLC player has streaming features built in and using RTP streaming protocol, a local video was first streamed from one PC to another PC running the player. Another 3rd party application, 'Mobiola' software, was installed on both mobile and PC to open a connection from the mobile's video camera to the Mobiola software on PC. The video stream is then captured as if it is an external web cam and was streamed with the above process to another PC.
This idea was used as one of the bases to achieve the live streaming feat. This will be explained in detail in the following section.
Video Streaming on a website using HTML5
One thing that needed to be considered for playing the video stream on an internet website is using a video web streaming player plugin such as Adobe's Flash or Apple's Quicktime. Adobe Flash Player is still widely used to embed videos on websites such as Youtube and similar sites.
HTML5 video is a new way of embedding videos online on an HTML page. This eliminates the need of external plugins for playing video streams.
Video formats that are supported with HTML5 are H.264/MPEG-4 AVC and open Theora and OGG formats.
To start with, MIME types for Theora video were set on our server so as .ogg files can be played. This is done by adding the following code:
AddType video/ogg .ogm
AddType video/ogg .ogv
AddType video/ogg .ogg
AddType video/mp4 .mp4
An H.264 and Theora video succesfully played on the site after they were manually added to our server.
A connection with a mobile device and VLC software was done as explained above and this time an iPhone was used. After a connection between mobile and the PC was established, VLC software first captures the mobile's video stream. Then HTTP protocol streaming was selected but as noticed in the screenshot below, we are not defining a specific address to where the video will be streamed. This is since the video will first be streamed locally on the same PC. The video transcoding for Theora + Vorbis (OGG) was selected so as it could be played on an html site further on. On the same VLC software, we modified the "Generated Stream Output string" path so as to include a filename.ogg after the port number.This is done so as the web browser will know how to handle the file extension.
When the following code http://localhost:8081/stream.ogg was entered on Mozilla Firefox Web browser, the video stream was determined to be working.
The next step needed is to use the video being streamed and embed it on our APT website. This was successfully done by implementing the following code on the html website:
<html>
<head>
<title> GAPT HTML5 video Live Streaming</title>
<!-- This is the code to enable HTML5 video tags in the website; no plugins needed -->
<script src="http://html5media.googlecode.com/svn/trunk/src/html5media.min.js"></script>
</head>
<body>
<video width="640" height="360" controls>
<source src="http://localhost:8081/stream.ogg" type="video/ogg"></source>
<! -- Gets VLC http stream --!>
</video>
<p>Direct Link to view stream: <a href="http://localhost:8081/stream.ogg">Live streaming"</a> </p>
<! --This is another option to connect directly with LocalHost on PC -->
</body>
</html>
The above code connects the video stream directly to the website. Since this process is done on the web server itself, the video stream can be viewed from any PC connected to the internet or to the same network as the server.
The above Screenshot shows the system being implemented with VLC player on the right and on top of Mozilla Firefox web browser.
Conclusion
Suggestions for further work
Website:
As can be seen the backbone of our website is quite simple given that the main focus of the project was on the interaction of a user-pc with the mobile. Built solely for the purpose of acting as "the middle man", the website possesses virtually no artistic creativity. Therefore this aspect of the website could definitely be given more attention in order to make the website more visually catchy whereas we built the structure of the website with the only purpose in mind to make it user-friendly.
Python:
Video Capturing: The length of the video captured is pre-set to 10 seconds in the implementation of our project. In an ideal situation, we could have allowed the user to input a value for the timer from the website and send it as a packet along with the string of characters "video" when the request is sent
Live Streaming:
Our live streaming feature works via VLC software on the PC. (VLC is streaming the video on the web server). What could have been done is that a program could have been written to initiate the video stream directly from the mobile device. To enhance the server capabilities, a type of streaming server like Darwin could have been implemented for improved streaming quality.
Applications:
As a project itself, we see a lot of potential in further development of the prototype we have worked on during this semester. At one point in time during testing, a request was sent to a mobile to capture a video while the mobile was preoccupied with a phone call - the phone call was uploaded to the website with both parties at either end the call sounding crystal clear. With this being said, perhaps further research could be done with the possibility of part of this project being used to record telephone calls. This project as a whole could also be used as a means of communications between family members and friends who live abroad by keeping an online journal of one's life away from home via pictures, videos and the occasional live footage when one side is on the website at the same time the other side is sending a live stream. Another possible application for such a project, given the appropriate further development is the monitoring of a parking system where a camera would capture a photo of an approaching car and evaluate its number plate. Whilst going through a database to find the mobile number of the respective owner, the cameras monitoring the parking area would locate an empty parking space and the system would send a message to the driver to alert of the vacancy.
About three months ago we were faced with a challenge for which given our academic background (all coming from a Mathematics and Physics background) the research and orientation phase played a very important role. Looking back, the most intriguing aspect of this APT was without a doubt the amount of exposure we gained to several new environments, particularly Python, HTML and PHP. The amalgamation of these three programming languages helped us open several windows of opportunities until we reached the final version of our prototype. Most importantly, we got a lot of hands-on experience in programming, mobile applications and building a website - three skills which are not only desirable in our line of work but as very interesting skills that have the potential to give rise to several creative outlets in the technological industry. Apart from the academic achievement obtained through this project, one must not forget to include the value of time management that also played an important role in the execution of this APT. A good balance between initial research, development, testing, and finally organising everything and bringing the project together made for the success of this project which ultimately exceeded our expectations.
