Sunday, December 17, 2006

python map vs C map.

Ages ago I made some python implementations of map.

Map allows you to call a function on every element of a sequence.
[2,3,4] == map(lambda x:x+1, [1,2,3])

One implementation I made used threads. Which made threaded programing easier for myself.

However the interesting thing was comparing the speed of the builtin map to the map function accelerated by psyco.

It turns out that for a whole bunch of cases the psyco version of map is faster than the CPython version.

Here's some unittests and the map function which show the speed differences.
http://rene.f0o.com/~rene/maps/map_unittests.py
http://rene.f0o.com/~rene/maps/maps.py

It shows how nicely loopy code can be speed up with psyco.





Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - python, php, mysql, e commerce, javascript, CMS, css, flash, actionscript, games, postgresql, xml.

Sunday, December 10, 2006

Galcon released.

Risk with spaceships. Galcon is kind of like the game risk - but with space ships.

The most fun can be had playing multiplayer on the internet with others.

Download the windows version


Download the MacOSX version

A linux version is also in the works. It should be up on the site within a week or so on the Downloads page.


Galcon was made using Python, pygame and PGU. It uses some C extension modules for the graphically intense parts. Like the additive blending modes for particle systems, and custom nice fast space ship drawing code.

It's great to see another quality game being made in python.

The networking uses UDP so that network games are quite good. I can even play games here in Australia with people in the USA and elsewhere. Even with 400ms ping times.

The single player game is fun too. With a whole bunch of different missions to play through. Different AI bots are used which give different challenges.

There's a small group of Galcon multiplayer people giving the game a whirl most of the time. Some of them ranked as Admirals, and others ranked as lowly cabin boys.






Congratulations to Phil for releasing Galcon! I know Phil has worked quite hard since Galcon won the ludumdare game development competition six months ago. Six months of polishing Galcon so that is plays really nicely has paid off.


I'll see some of you in the multiplayer battles for Galactic Conquest. Attack!!!




Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

Friday, December 01, 2006

Batching as applied to websites.

I have 10 pages of content. Each of those pages is complex, and will most likely be visited by the person when they arrive.

40 images are on some pages. 130 small images all up. The images don't change that often. So after the first visit they are not downloaded again. With well specified expires, and cache directives sent out by the web server - the browsers don't even check to see if the images changed all of the time.

However a browser may only allow four requests at a time. This means it's going to take a while for all of the 130 images to download.


The obvious solution is to batch all of the requests into one. Since web browsers are dumb, they don't do this. HTTP was poorly designed in this regard. They didn't allow batched downloads. eg 'GET image1.jpg,image2.jpg,image3.jpg' or something similar is not allowed with HTTP.

Multiple requests in the same connection were not a good idea. Since there is latency involved for each file requested.

So one hacky way around this is to batch all of the content together yourself.

Using code and algorithms I developed for batching textures with OpenGL programs, I can combine images into one big image. Then you use css to set clipping areas, width and height of each image - so it only shows the portion of the large image it is supposed to.

For the html content, I encoded it into a javascript file using a json encoder. Then constructed the pages with javascript.

The result? 140 requests down to two requests. That is a 70x improvement.

As you can imagine, this results in much faster browser rendering of lots of data. Meaning that I can put a lot more on a web page than with a non batching method.


I wrote an article on Batching Apis.





Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - python, php, mysql, e commerce, javascript, CMS, css, flash, actionscript, games, postgresql, xml, asm.

Friday, November 24, 2006

gvim - opening files in tabs. How to search through windows?

With gvim 7.x you can open files in tabs.

However on the command line it follows the old behavior of editing one file at a time.
eg:
gvim file1.py file44.php file2.py

Instead you have to use the -p command line option.
eg:
gvim -p file1.py file44.php file2.py

Then you have all of your files opened up in separate tabs.


The new tabs in gvim 7.x make using pypanel less necessary for me. pypanel is a little python app for X11 that works really nicely with WindowMaker (the minimal window manager). It gives you a task bar, which can be programmed in python. I used to use pypanel with gvim for when I have more than a few files opened in one work space. Now with tabs, I am able to have a lot more files opened at once.

Gvim tabs, the pypanel task bar, and windowmakers workspaces make my desktop a lot more scalable, and less annoying :)


Does anyone know of a tool to search through window names, and then focus the selected window? Now that I might have 100-200 files open at once a search function would sometimes be handy.


Another fun gvim trick for my Work In Progress (WIP) todo/log files is to insert the results of a command into your file. :r!date
Thu Nov 30 10:29:00 EST 2006

Now I don't have to leave the keyboard to insert the date.






Melbourne Website DeveloperThis post written by a Melbourne website developer. I am available for your web development projects. My skills include: python, php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

Compiling pygame on windows with MSVC

John Popplewell has made a guide for compiling pygame on windows with the MS visual C compiler.

http://johnnypops.demon.co.uk/pygame/index.html


Looks like a pretty good guide to compiling pygame on windows. It shows you how difficult it can be. With unix it's much easier to compile.

You can also compile pygame with mingw and msys using gcc. It's equally as complex - just in different ways.


Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

Friday, October 20, 2006

GET urls with side effects can be a security problem.

GET urls with side effects can be a security problem.

For example, imagine the following urls:

paymoney?user=joe&amount=3000.0
deleteSomething?file=important.xls

Now if those urls are protected by a login system, then only those who login can use them right?

Nope. It is easy enough to trick someone into visiting a web page so that those urls are called by their web browser. If that person is logged in when they are tricked, then the action happens. A person can be tricked either by using img, frame, redirect or some other tags or http/html/flash/javascript features.

This is because according to a web browser it is ok to include or link to elements on other pages. In fact that's the whole point of hyper linking.

In this way it uses the authorization of person viewing your well crafted page. You can now create a page so that you can delete files as someone else, or pay any amount of money you want to anyone. Whatever the badly designed GET urls allow you to do.

This can even be done with POST. However it's a little harder.

So lay off the GETs with side effects.

This problem is called Cross-site request forgery




Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

Wednesday, October 18, 2006

CYMK coming to a gimp near you?








I saw this in the latest development release notes of gimp 2.3.12:
"- build a color-managed CMYK color selector if lcms is available"
http://developer.gimp.org/NEWS



So it seems the gimp is getting CYMK support(or has it already). That'll be good for those doing print work that requires it.

Along with the gimps next generation image processing core (GEGL) things are looking good for the gimp. http://www.linux.com/article.pl?sid=06/10/16/1342216 http://www.gegl.org/

I really think the new versions are quite good. With most of the bugs that annoyed me gone, and with new features being added all of the time.

There's still some font handling issues which would be wonderful if fixed. Like rotated fonts, and fonts that can follow a path. However I think those features will appear over time.

This line may be interesting for those python users out there wanting to optimize their image production pipeline... "- many improvements to the Python bindings and the pygimp user interface"

Fetch the gimp. But the gimps sleeping. Go wake it up!



Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

Saturday, October 14, 2006

Making fonts on linux.

To design a font on linux you can use fontforge. http://fontforge.sourceforge.net/

As part of my learning graphical design, I have become interested in finding out how fonts work and how to design fonts. I think learning to design a font will give me insight into other fonts. Much like how learning assembly language gave me insight into how computers work.

Creating a font on linux with fontforge is the way to go I think. I haven't found any other way to create a font on linux yet! It was quite hard to find fontforge for designing fonts on linux too.

It's a time consuming thing - designing a font. That I think will take years to figure out. I don't expect to make a great font, however I think there is room for creativity in fonts yet. So I hope to make something useful and learn something on the way.

I guess I have made bitmap fonts before including the animated one on f0o.com http://f0o.com/. However crude it might be ;)

True Type Fonts .(ttf) fonts contain a virtual machine, much like postscript. With it you can instruct how each character is rasterized. So to make fonts well you need to be able to program this virtual machine. So like many other areas of graphics - a knowledge of programming can help you to design a good font. There are some tools for working with ttf 'hinting' instructions here(including a compiler from a C like language): http://home.kabelfoon.nl/~slam/fonts/

Thursday, September 14, 2006

Python pickle and web framework security.

Some python web frame works are using pickle to store session data. Pickle is a well known poor choice for secure systems. However it seems to be more widely known by those writing network applications, than those making web frameworks.

Is your web framework using pickle for sessions despite the warnings in the python documentation about it being insecure?

By using sessions with pickle people who can write to the database servers session table can execute code on the app server. Or people who can get data into the session file/memcache data store can execute data.

This might be an issue if the database server is run by separate people than the app server. Or if the session table is compromised by an sql injection attack elsewhere.

There are some more secure ways of storing pickled data.

Pickle is deemed to be untrustworthy for data. In that it is not certain that code can not be snuck into the data that will be executed by pickle. So if some data from user input is put into the pickle, then it is possible that code could be run.

There are some people who know more about how to exploit pickle, however the warning in the python documentation is this:

""Warning:
The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source."""


Cerealizer might be an alternative option...
http://home.gna.org/oomadness/en/cerealizer/index.html

Or maybe these other two.
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/415503
http://barnesc.blogspot.com/2006/01/rencode-reduced-length-encodings.html

Wednesday, August 30, 2006

x86 the VM.

It's been really interesting to see VMs over the last couple of years.

Now there are emulators, and virtualisers which are capable of running x86 really quickly. The processors themselves don't run x86 natively anymore, it's a VM.

Now Apple are using x86, and x86 is getting more common in the embedded world too.

So now, rather than creating a VM like python does it seems to make sense to use the standard VM, and that is x86. Of course x86 is really complex, and still fairly slow to emulate on slow hardware. So using a simpler VM still has its advantages.

However writing directly to the most common VM has its advantages too. You can make software which is 400 bytes big which can do almost the same as a 8000 byte program. That's a 10x saving in program size. The same program will run in 12KiB of memory, instead of 1.7MiB of memory. That's a 141x memory usage saving. Because the code size, and memory size is so much smaller you can get a lot more done with the same amount of memory. You can run an entire OS and programs in less memory than a python process uses on debian linux. You can also use a lot more of these tiny processes. eg. for a single purpose webserver you can handle 40,000 connections using fork and separate processes in around 640MiB of ram with a duron 850. Or 100
connections using 1.6MiB, which is less than a single apache process.

I find it very interesting, and think that assumptions about processors and architectures that I learned 10 years ago are perhaps changing. The standard thinking is that fork is slow, and that you should use event driven async for high speed. Well it's not slow if your processes are only 800 bytes worth of code. Just enough code to do the exact task at hand.

Communication between processes has become really quick now. With system calls on linux like slice, tee, and vmsplice you can get two sockets transfering data directly without having to read data into user space. So you can take the output of a mpeg encoder and send it directly to a file on the HD not even copying the data, at the same time send it out on a socket through the network card. All without going into user space. How about compressing a stream of data, and sending it to a socket and also saving it in a file cache.

Processes and VMs can be transported between machines. Mosix can move processes amongst machines. With emulators you can store the whole state of the system in a file. You can store your x86 system image in a file, and move it to a different machine.

Then there's all this GPU general purpose computing stuff. You can run batch processes at 100x the speed of a cpu on these things. There's still a lot of science being done to find good ways to reimplement algorithms that work well, but lots has already been learnt. With the internet I think they have progressed much more quickly than in the 50s-70s. Chips like those found in the GP2X have multiple cpus, DSPs, and all their IO devices all on the same chip.

- tiny programs. http://asm.sourceforge.net/asmutils.html
- emulators and virtualisers http://fabrice.bellard.free.fr/qemu/
http://www.thefreecountry.com/emulators/pc.shtml
- mosix, distributing processes. http://www.mosix.org/
- general purpose computing use of GPUs http://www.gpgpu.org/
- tee, splice, and vmsplice in linux 2.6.17 http://kerneltrap.org/node/6505

There's lots to be excited about with the new ways people are approaching VMs and computing.

Monday, August 28, 2006

pyweek 3 theme voting has begun.

http://www.pyweek.org/3/

The theme voting has begun. Here are the themes for the week long python game jam.

* Pick a card, any card
* Watch me pull a rabbit out from this hat
* Sawn in half
* Spoon bending
* The Disappearing Act

A very short list of themes to choose from this time.

Thanks to Richard for organising pyweek again! If you have some spare time coming up, and want to finish a game then you should enter pyweek. If you don't have much time, find a team and join that.

Let's hope the server stays up mostly :) These competitions all ways result in a server going down. Lots of users, and wide spread attention amongst smart, young and demanding hacker types means it gets challenges lots of other websites do not get. Strain on the software, and on the hardware are two of the causes. Otherwise it is caused by a simple cracking in to the machine to take it down.

Thursday, August 24, 2006

Qwerty rhymes. With Flirty.

not a bit of good food
for got home gut
jill had her kill


Oh, how I love bad poetry. I just had to share. You need to type it to appreciate how bad it is.

Saturday, August 12, 2006

Django security.

I replied to a blog post by Jason Huggins on his blog asking for a web frame work evaluation check list. I listed a number of common security issues, and finished my post with:

"I can not see any python web frameworks that do not have a history of security problems. I also can not see one which was designed from the beginning with security in mind."

Then James Bennett (a Django contributor) asked about problems with Django specifically. Saying 'if there’s some long bounding history of security flaws in Django, I sure haven’t seen it'.

http://www.jrandolph.com/blog/?p=45

Unfortunately my reply seems to be caught up in the moderators queue, so I guess I'll have to put my answer below. A couple of posts after mine are there, so I am not sure why my post was not approved to be posted.

ps. since my post on the 11th of August I have reported four security issues to the django project. Not one has even been replied to. Not even with a simple response like 'I got your email, we are looking at it'. It's great having a policy, but if you don't follow it then what's the point?
UPDATE: Adrian has been notified, and is looking into it. There was a problem with the email alias.

Have fun!





# Rene Dudfield Says: Your comment is awaiting moderation.
August 11th, 2006 at 3:37 am

James. You’ve answered your question about the Django security history for me. I can not see a list of security incidents listed on the Django site. I think a good idea would be to list them on the front page in fairly large type. However those ones you mentioned are only a few months old, so I reckon there’s more problems.

For XSS you need to start with white listing, and go from there. There’s a bunch of good articles on the subject. Just escaping stuff is not enough.
http://www.iamcal.com/publish/articles/php/processing_html_part_2/

http://cr.yp.to/ is a good source of information about writing secure unix network software.

If you assume that user submitted content can not be cleaned, then is the system still secure? Feedparser, with 3000 unittests and heaps of high traffic sites still got it wrong.

Not only is Django itself not secure, but the pieces it is built on are not secure. Just search for apache, mod_python, and python security issues.

If you are sure about the current state of Django then maybe put up an offer for $1000 if no holes are found ;) I’ll give you $1000 if Django makes it through the next year without a hole found.

Jason. I’m not sure of a web framework which has security considered in the design from the start. It’s really good that the ruby developers have started to put an emphasis on it. I’m going to take this oportunity to evaluate my own security short comings, and I’m glad that others are too.

Saturday, August 05, 2006

Pyweek 3, one week game jam in september.

The third pyweek competition is starting in September.

You can have group entries, or go solo. It is a competition of sorts, but it is all just for fun really. You have a week to work on your game. Most people work on it in their spare time. Other people take a week off to do it.

http://www.pyweek.org/3/

Tuesday, July 04, 2006

python2.5. faster startup, better memory use, faster.

Recently I was testing that pygame still works with the python2.5 beta. When I was doing this, I had a chance to play around with python2.5 and all it's new goodness. Amongst the other good python2.5 features is that python is performing better.

Almost twice as fast a startup 'cold'.

This is because it uses half the number of syscalls as python2.4 does. 755 syscalls in 2.4 and 461 in 2.5. Still not quite as good as perl with 40ish syscalls, but a massive improvement none the less.

$ time python2.4 -c "print 'hello'"
hello

real 0m0.923s
user 0m0.041s
sys 0m0.025s

$ time python2.5 -c "print 'hello'"
hello

real 0m0.417s
user 0m0.043s
sys 0m0.021s

However compared to perl, python is still syscall heavy. Perl does around 50ish syscalls to start up.


Python2.5 also frees up more memory that it isn't using.

Python2.4 does not release all that much memory if you use up quite a lot. It holds onto it in it's memory pool as an optimisation. The idea is that it is quicker to reuse memory, than to rely on the operating systems memory management. Unfortunately, this can be *really* bad for long running processes, and on machines which need to share with other processes. ie webservers, games, gui apps.

Even though there could still be improvements, my tests with real world applications using 2.5 show a much better use of memory. It gives back most of what is used. However for best use of memory it is still a good idea to restart processes if you can.

Monday, June 19, 2006

Debian libc6-i686 for faster python.

If you have a non 64bit Debian machine (like a duron, p3 etc) installing libc6-i686 will speed it up.

# note that doing this you may need to restart running programs otherwise they might crash.
# rebooting might be an idea, if you do not know how to restart all your programs.
apt-get install libc6-i686

After I had it installed I got a 7% faster pystone result. I also got a couple of fps quicker in some pygames. This was with an AMD duron 850mhz system, your milage may vary.

Wednesday, June 14, 2006

Mozilla firefox binary faster than debian binary.

It makes sense I guess. The people who make firefox should be able to make a faster binary than a distro.

The mozilla firefox build is i686 rather than i386, and probably uses some other methods for extra speed that debian doesn't. Debian stuff is usually made for stability and compatibility first, rather than for speed.

However, so far on my machine the mozilla build has been running for a day or so. So it works nicely at least on my machine.

It's made my browsing noticably faster on this duron 850mhz machine that I use daily. So I'm quite happy about it. The windows refresh faster, and the javascript heavy sites do their thing quicker. My pretendpaper site loads a whole 1.5 seconds faster using this build. It's probably the most slow, and intense javascript based site I know of. The load time is noticably slower on a slow cpu machine.

Pygame moves to subversion, and more 1.8 updates.

Pygame has moved to subversion. Thanks to our wonderful hosting friends at seul.org for setting it up for us.

The sdl ctypes has been moving forward. This is a project sponsored by the google summer of code. A couple of SDL example programs are running, and the code is looking good. There is a manual, and a couple of tests. sdl ctypes is being developed in the pygame subversion. http://www.pygame.org/ctypes/

For pygame I have gone through most of the documentation comments on the website. Only 20 remain left to sort through. shreadwheat also added some more documentation enhancements.

A request for the scaling functions to be able to accept a destination surface has been made. I've mostly completed this change. It allows for faster scaling when you are repeating the scale operation multiple times onto the same surface. This is because you don't need to allocate memory so often. However the destination surface has to be the correct size, and format. If it isn't an exception is raised letting you know.

This brings up a point about a surface pool. That is rather than create new surfaces, ask to get one from the memory pool. If there are any left in the pool give one of those rather than creating another one. Once that surface is not referenced it will go back into the pool. This is a common pattern to increase speed where lots of allocations are made. For some programs using pygame that create lots of same format surfaces repeatedly this should be a good speedup. Especially for larger surfaces (50x50 or bigger).

I'm probably going to give pygame another month before feature freeze because of work commitments. So there should be plenty of time to add anything else to pygame if you want it in there.

Google earth for linux will bring opengl for linux?

Will the arrival of google earth mean that hardware accelerated opengl be more widely spread on linux?

It's a free fun toy that requires opengl. Hopefully it will do more for linux opengl than quake3, and xgl have done.

I hope so anyway.

Sunday, May 28, 2006

Pygame 1.8 release gets closer.

The last pygame release was August 16th 2005. So there are a lot of bug fixes and a few features in development that need to get out there into a release. With the new SDL being released last week it seems like a good time to get a new pygame out.

Here is the release plan from the last release. We will follow it in the same way again for this new release.

At the moment we are in the apply patches, fix bugs, get in any last minute features phase.

This weekend I made pygame.image.save be able to save as .png and .jpg files. Since pygame already optionally links against libpng and libjpg to load .jpg, and .png it shouldn't be too much hassle for people, and only a tiny bit of extra code. So if pygame is linked against libpng, and libjpg it should be able to write .jpg and .png images.

Saving as jpg/png is an often requested feature for people doing screen shots, caches, and level editors amongst other uses. Hopefully these save functions will eventually make it into SDL.

That's the last feature that I want to put in pygame for the release. Microphone, and line in support will have to wait for SDL 1.3. Since a sound api change is needed to work it in nicely. See bug #10 in the SDL bugzilla 'audio input API'.

I've been slowly going through the doc comments on the pygame.org websites documentation. This is where anyone browsing the documentation can make a comment on each function, class or module. There were around 100 or so comments from people. Ranging from 'hey there' to useful tips, typo corrections, documentation clarity ideas, and bug reports.

Since a misleading or unhelpful comment is worse than a code bug, it is very important that these comments get worked into the main pygame documentation.

However since some comments are bug reports it might take quite a bit longer than I thought to work through all of the doc comments.

Saturday, May 13, 2006

Best and easiest approach for CPython speed ups. Processor specific C modules to distutils. mmx, sse, 3dnow etc.

For pygame which uses SDL some parts are written in assembler. These parts detect which cpu they are running on and then use those mmx/sse/3dnow optimized assembly routines if those processor features are available.

However much of pygame and SDL is still in C, not ASM. So compiling using processor specific features does give a very nice speed up for the C parts.

For some parts a 33% speedup or more can be gained. Just by changing compilation flags. I think this is the best and easiest approach to getting parts of CPython to be sped up. Below are my notes, and thoughts about the topic. Mostly this is based around my experimentation with pygame compilation. This does not take into consideration python compilation itself, or any of its modules. However a similar methodology could be applied to speed up pythons modules too. Note that recompiling python itself to match your processor can give a large speedup too.


So, I have started experimenting changing distutils to compile modules multiple times with different flags for processor specific things. eg. amodule_mmx.so amodule_sse.so etc. This is just for pygame at the moment, not for python modules in general.

I am thinking of doing three or six sets of modules... to keep the disk usage down. An athlon 3dnow version, a p3 SSE version, a pentium mmx version, etc. All of the pygame C modules are around 430KB. However if I only do the ones that are generally cpu intensive it goes down to 230KB. So an extra 690KB to 1380KB uncompressed. Compressed it is about 348KB to 696KB extra.

So if I add that to the windows installer, or to the .deb packages it's not too bad. For people who compile it for their machine I'm thinking of just making it compile only the one which matches their machine. Which means I'll need to put cpu detection code into the compile phase.

For people who distribute their own py2exed version, they'll have the option of including extra cpu specific modules, or keeping their download a little smaller. Or even including more cpu specific modules if they want.

However I still need to finish the cpu detection code. I'm going to base it off SDL code. Since pygame requires SDL anyway, and I'm sure that code is widely tested. I'm thinking of having a python wrapper function which detects the cpu features, then tries to import the relevant processor specific .so. Then it keeps trying to import ones best first until it finds one. So if no processor specific ones are found, then it uses the default one.

Another issue to think about is if specific cpu instruction sets give enough of a performance boost or not. So if pentium mmx is almost as fast or faster than P4 SSE2 code, then I might as well not include the P4 SSE2 version of the module. This will require profiling to figure out.

So profiling more of pygame is needed to get better results. For example a script which loads lots of jpg and png files. Something which blits lots of stuff. etc etc. Each of these profiling tests should output timing data in a standard way. So that they can be run automatically, and then submit their data. I think I'll setup a web page to collect this data. So if people are able to help, they can select to submit data from their machine.

Also needed is better automated testing coverage of pygame. So that I can test that the recompiled executables run correctly. This is especially needed since python uses experimental compilation flags on some platforms. ie gcc -O3. -O3 has some known to be potentially buggy optimizations included. I have come across situations where compiling python extension modules is buggy because of this use. Also some of the processor specific optimizations are not as widely used or tested.

Another downside to this is that distutils changes with python versions. So any 'monkey patches' to it need to be tested with new python versions. This needs to be done for windows pygame anyway. Since it patches distutils for the Mingw MYSYS compilation environment. Eventually once this technique is perfected patches will hopefully make it back into distutils of python. Meaning less work for each python release. However since pygame needs to be recompiled, and tested for new python versions anyway, this isn't a major issue.

The funny thing is though... I'm not even sure if the python distribution on windows works with older cpus. Since they have compiled python with the newer C library. This means it won't run on some older computers without them getting the C runtime. That's with py2exe versions of the programs anyway.

This is just going to be for x86 win/*nix/*bsd machines using GCC for now. Not for Mac machines, because I don't have one to test with, and the binary situation there is already weird enough. I will also not use the intel, microsoft, or VectorC compilers to start with. Even though they are better at some optimizations than GCC. That is an exercise I'll leave until later.


Hopefully this should allow pygame users to not worry as much about optimization. Or at least people will be able to put more sprites on the screen before worrying ;) It will also make many existing games run faster or use less resources whilst running.

Thursday, May 04, 2006

Pygame, Blending blits, mac and windows clipboard/copy paste working.

An update of what has been happening recently with pygame.

Thanks to a kind person on the mailing list the macOSX version of the clipboard code was tested. It was a long process but eventually. A one week debugging cycle because I don't have a mac. HAHA. I also got around to testing out the windows version of the code on my win XP box. It worked fine there.

Some blending functions have gone into CVS. So you can blit with the modes ADD,SUB,MIN,MAX,MULT. This allows more you to do lighting type effects, and is very useful for particle systems. These are the missing things that SDL doesn't have which are often asked for. The code was based on Phil Hasseys code which he wrote for 32bit software surfaces. He's also done some other code for lines etc. So I'll want to incorporate those changes so you can draw lines that blend too.

I also need to add those blending modes to the surface.fill() method. That'll be very useful for fade to black/fade to white. Or fading text in and out.

There have also been a few patches submitted, which I have done unittests for. Pygame doesn't have too many unittests. But now I am creating one each time a change is made. To test that the bug is actually fixed, and to test if the feature added works right.

Oh, one interesting point of note. I used the TinyCC for my pygame development. It can interpret C code! How cool is that. It is also heaps faster at compiling than gcc. So for development it's really nice for when doing C code. It's even faster at interpreting code than python is. So no more slow compiles for me!!! Tinycc apt getable too.

TinyCC also has a library which can be used for code generation. I reckon weave might be very nice with this :)

However gcc is better for finished code because it generates faster code and is more complete. But yah for rapid prototyping!

Favourite student slave labour tasks for python.

These are my favourite python ideas from the google summer of code.

Ones that I think will benefit the most people, and affect the most change for the better. Ok, just ones I reckon would be cool.

  • Implement ctypes support for GCC ARM platforms. The underlying issue is lack of closure API support for ARM in libffi. A patch available at [WWW] http://handhelds.org/~pb/arm-libffi.dpatch, that should be hopefully a good starting point. ctypes CVS has a libffi_arm_wince directory, which also seems to support closure API.
  • Make a python+pygame plugin for IE, netscape. CodingProjectIdeas/PythonWebPlugin

  • Psyco for MacOSX. PPC, and universal binary versions. (also psyco for ARM would be cool!!)

  • Research how to get python support into all the cheap webhosts.

  • Security audit of python. Using as many automated processes as possible.
  • Python speed up. Reduce memory usage, speedup startup time. The two main speed regressions of the 2.0, 2.1,2.2,2.3,2.4 releases. (438,453,499,771,880) syscalls vs 106 for latest perl. 0m0.031s, 0m0.029s, 0m0.037s, 0m0.059s, 0m0.057s real time to start vs 0m0.007s for the latest perl.
  • Code coverage. (there are a few suggestions for code coverage).
  • Improve thread performance. (reducing memory usage will probably help thread performance the most)

The memory usage, and start up times would be great to improve. As I think these are the things which make people avoid using processes. Processes give us protection, and are a great way at keeping parts separate. However if processes are so expensive to use in python, then other means will be used instead.

A security audit would be wonderful. I'm not sure if a security audit has been done on python, and I'm sure it would lead to a bunch of bug fixes. So a security audit is much in line with using code coverage of tests. I guess a lint, and pychecker would be other good things to add into the python build process.

Friday, April 07, 2006

What is next for pygame?

Last week I checked into cvs a 'scrap' module. Which is basic copy/paste clipboard support for windows and X11. So you can copy things to and from the clipboard. It is based off the scrap code found off the SDL site.

I still need to finish wrapping a couple of C functions, but it is almost finished. It has only been tested on X11 so far. However the original code was written to work on windows as well. So hopefully copy/paste will work there too. Then left to do is Mac OSX support.

Today I did some code today on getting TinyGL to work with my character animation code.
I managed to get a model animating, and at a decent frame rate. Of course there was no texture on this model. Which is the slow part. However even non textured 3d stuff is pretty useful done in software quickly. I did convert one simple texturing demo to work with TinyGL, and it ran ok. So I am hopeful for its performance.

TinyGL only has about 100 functions which need wrapping. Most of the functions have been wrapped with pyopengl1. PyOpenGL1 was the one which didn't use swig.

I am going to see how fast it runs with texturing before starting on wrapping it for python.

Wednesday, April 05, 2006

Stable - Api. Does software still work from three years ago?

I saw yet another 0.x piece of rapidly changing python code calling itself stable today.

If your code meets any of these criteria your code is not stable.
  • Is not even six months old, with a 0.x version number with massive changes planned.
  • Your code is still old and changing rapidly with massive changes planned.
  • Your api is changing causing breakages.
A good test to see if your library/framework is stable is to take some old program and see if it runs correctly, as it used to. If it doesn't, then your library/framework is not stable.


ah... words that have different meanings.

Tuesday, April 04, 2006

Zanthor. Our pyweek2 game.

Zanthor is the game we made in one week as part of the pyweek2 competition. The idea of pyweek is to make a game in one week mostly from scratch.

You need python and pygame to run our game.

You can download it here:
http://www.madecollective.com/~rene/stuff/zanthor_source.zip

It works on Windows, Macs, linux, freebsd, and anywhere else python and pygame run. You can play with keyboard, mouse, or gamepad.


Here is a screenshot:



Please let me know if you like it, and of any cool ideas for it.

Monday, April 03, 2006

Melbourne International Comedy festival.

The Melbourne international Comedy festival is starting soon.

My comedian friend is doing a show for it again. He has made a sudoku solver using an Excel spreadsheet. For something fun to put on his website.

I thought it would be an interesting challenge to write one in python.

Unfortunately there already appears to be 120million different sudoku solvers already written in python so I gave up. Such is life. After the pyweek2 finished I'm a bit shattered, and programming tasks seem a little hard. I need more sleep.

If you're in melbourne, you might want to check out his comedy show: http://www.grouphug.com.au/tmr/. Last years one was quite funny.

Sunday, March 26, 2006

Pyweek 2 begins. Concept sketches done.

Pyweek 2 has begun. With the "it's powered by steam" theme winning the vote. 190 people have signed up to compete. Not sure how many people voted for themes. Hopefully quite a few people will actually compete.

So here is one of our concept sketches that Aaron - aka did.


We have a basic web server code done. Using wsgi, twisted.web2, and sqlobject.

Since the game is going to have elements where you can affect the worlds of the players from a web page. So even if you don't play with the game, you can have fun messing around with other peoples game worlds.

However the web element is going to be an optional part of the game. The game needs to be playable in single player mode.



We also have the start of our client main game done. Using pgu, and an isometric game view. You will be able to play as a human who controls a robot castle which is powered by steam. We are currently figuring out what gameplay, and plot we want. But have the basics of what we want to do sorted. Enough to make a prototype.

Game play wise we want it to have a few goals that are apparent for the player. A 30 second goal, a 5 minute goal, and a 30 minute goal. We also don't want to do anything too cliche, or traditional.

So we have a place holder iso level being displayed. We also have the sound code basics there thanks to the rdpyg library.

Tonight I am going to get some of the basic gameplay stuff in whilst the US guys sleep.

You can see pyweek 2 diary entries for all of the other s who are posting them here.

Wednesday, March 22, 2006

Five themes for the pyweek competition.

Here are the themes for the pyweek game development competition.

* Someone else's trash
* A fraction too much friction
* Mind the gap
* Doorways
* It runs on steam!

Much more interesting than last pyweeks abstract ones.

It runs on steam! is my favourite so far. It puts the most images into my head. Images of Victorian era steam races, and of a future world which runs on steam powered technology.

Saturday, March 18, 2006

PGU play as preparation for pyweek.

Today I did a little test app with PGU and pygame in preparation for pyweek. I have not played with PGU for a while, so thought I needed a little refresher course.

PGU works with pygame, and has a gui library as part of it. It's a cross between html, and QT. It even has a basic html renderer. So if you know a little html, you can use some of that knowledge.

The result is a simple RPG character attributes generation widget character_create.py





It can generate these character attribute scores, and allow the user to assign points to them. I made it reusable so that people can plug in their own character attribute names. Eg - you could have 'Sense' instead of 'Dexterity'. You can put these widgets inside other containers, so that you can generate multiple characters at a time.

Our pyweek team The Olde Battleaxe is going to do a few more practice sessions before the competition begins next week. Our team has one sound+music guy, a graphics+sound guy, and two coder+gfx+sound guys. Two of us are in Melbourne Australia, and the other two are in the US.

With the themes for competition being announced in ten hours or so we will soon be able to vote on themes.

Tuesday, March 07, 2006

Python is also good for games, so should be on python.org.

The new python website is up. It mentions web stuff, database stuff, and gui stuff. However it doesn't mention games or graphics stuff at all.

Links to pygame.org, and pyopengl.sf.net should be added to the front page. As they are very popular python libraries used for games.

Eve online, and the Temple of Elemental Evil are two retail success stories. The witches yarn is a shareware success story, and amongst the open source success stories are angry drunken dwarves, solarwolf, and pydance.

Blender, the gimp, poser, truespace, are amongst the graphics programs that use python for scripting. Making learning python scripting for artists very useful. ILM use python for movie making.

Tuesday, February 28, 2006

sqlobject transactions

When trying to figure out how to use transactions in sqlobject, I found this code snippet.

As a bonus my init database code now runs twice as fast. Yah!


def do_in_transaction(func, *args, **kw):
old_conn = sqlhub.getConnection()
conn = old_conn.transaction()
sqlhub.threadConnection = conn
try:
try:
value = func(*args, **kw)
except:
conn.rollback()
raise
else:
conn.commit()
return value
finally:
sqlhub.threadConnection = old_conn

# do_in_transaction(some_function_with_sqlobject_stuff_in_it)

# I needed to change my connection line to this. That is assign something
# to the processConnection thing in sqlhub.
# sqlhub.processConnection = db_connection = __connection__ = connectionForURI(config.conn)

Saturday, February 25, 2006

Pyweek registrations are open.

It looks like pyweek registrations are open.

Pyweek is a realy fun game development competition inspired by the ludumdare 48 hour competitions. Except this competition uses python, it goes for a week, and there can be teams or solo entrants!

0 minutes before the competition begins a theme is announced, and you need to make a game from scratch based on that theme.

The idea is to have a lot of fun finishing a game in a week. You don't work for the whole week on the game, just spend a couple of hours a day, and perhaps more on the weekend. So for some people this is better than spending a whole weekend.

For anyone interested in making games there is nothing more fun and educational than these competitions.

Lots of the fun is all of the energy and excitement you get from looking at other peoples work, and chatting away on irc. Everyone is making a game all at the same time, it's a lot of fun.

It is the game making equivalent of a jamming with people in a band. Except you are not in the same room, and you are doing it with up to 100 other people at the same time.

You can register on the pyweek site now.

A new version of Pretendpaper

After a couple of months of part time playing with the pretend paper code base I have put up a new version of this arts community site I work on for a hobby.

The most visible change is the modified css design. Which gets rid of black lines, and makes it look more minimal. These changes were done by Dane with help from Aaron.

A lot of the changes are not visible, as I rewrote things so that searching and editing are more doable. To make searching and editing how I wanted them to be I needed to be able to have multiple event listing tabs on screen at once. Which meant that I needed to change around the way it used ids to encode a tab id into each of the ids.

Also the code for creating new tabs stopped using function callbacks so much, and instead used objects. By using objects I am more easily able to change, and query their behaviour. I cleaned up the javascript code a little, but it still needs a lot of work. However a lot of it is still prototype code, so I'm not finished experimenting with it. Once the code is working nicely then I refactor it to work more cleanly.

Not that searching, and editing are finished yet. This version of pretendpaper was put up early so that we could more easily cover the adelaid fringe festival and have stuff for people to see before we finish the updates. The basics of one edit screen is finished. However it is more of a prototype edit screen. The edit screens need searching finished before they will be useful. As the edit screens require quite a lot of changes before they are done, and I'm still not sure how it will all look, or work out I am going to wait until the event edit screen is done before doing the other ones.

I think combining search in the edit screens will allow a more accurate and comprehensive set of data for the people looking at things.

It is interesting what things you do for a release that you don't do whilst not releasing stuff. So I think I'll continue with more releases

I spent quite a while debugging the session module I use so that it will work nicely with multiple threads, and multiple requests happening at the same time. I also made an in memory version of the cache that periodically saves its results to disk. It also saves its results to disk before the server is shut down. This is a nice compromise for me. The speed of memory sessions, with being able to save state, and be fairly resilient to power failures. The sessions are written to a temporary file first, and then moved over the other session file. Of course these memory sessions are not usable by multiple processes. For that I would need to use something like memcache. Or use the disk based sessions.

Some blog sumaries of artists are now going to be shown in the news sections. Only one blog is in there at the moment(Casionova), with more going to be added later next week. People still can not edit their own profiles, but when they can adding their blog feed will be the thing to do. Hopefully putting the blog sumaries here will make it easier for people to spread their message from their blogs to our site. Eventually it would be good to offer feeds of pretendpaper as well, but that will have to wait until the lofi static version of pp is done, and a million other things.

Because of a hard drive crash I needed to reinstall lots of software. All of the python dependencies that crept into pretendpaper made this difficult. I mostly use apt-getable python libraries, but because of various bugs I needed to install from source for many of them. This is the problem with using rapidly changing modules. I was a bit scared when I heard about SQLObject 2. However the changes will be welcomed, and api compatibility is planned. So I'm happy.

Saturday, February 11, 2006

Solarwolf game port for GP2X, and less sneaker net.

Yesterday I got some help from theoddbot on the latest changes on the gp2x scene.

There is now a serial over usb driver. So I can log into my gp2x and get a bash prompt through minicom.

Yah! much rejoicing. However the driver is buggy, and file transfers don't work. So it's still anoying to do some things.

My old development method, pre serial over usb. 'close down gp2x, then go over to win box, take out card, put card in reader, copy file over from linux to sd, then take out card, then put card in gp2x, then boot, then select program'.

Now I can make a bunch of changes, unmount the SD card through my bash prompt. Sneaker net them onto my windows box with the SD card reader. Then I can run the program, debug it, and make small changes with vi. I can also run code on the python interpreter.

Yah! If the program crashes, or gets stuck somewhere, then I can kill it from the bash prompt. However ctrl+c doesn't seem to work, so I need to run the program in the background to start with.


So after getting a few pygame examples to work nicely on the gp2x I began to get solarwolf running. With not much effort it was going fine. Now I am working on making it a good port. Making the buttons, and joystick to the expected things, increasing performance, and reducing memory usage.

One interesting thing, when I first ported the game sometimes it would get killed by the kernel on startup. This was because the game uses almost all free memory on the GP2x. I think the way that loading happens, sometimes because of the garbage collector there are times when memory is not freed right away.

There is hope in sight for using more memory... some of the reserved memory can be accessed through mmap, as well as by using hardware surfaces.

Wednesday, January 18, 2006

Angry, Drunken Dwarves


I packaged up Joe Wreschnigs Angry, Drunken Dwarves for windows today. Using py2exe and the NSIS installer.

You can download it here:
http://rene.f0o.com/~rene/angrydd_win_1_1.exe


If you play it please leave a comment on my blog.

update: it looks like piman has put the .exe on his page now.

Sunday, January 15, 2006

What I've been doing?



A few interesting game related things.

I have begun doing a series of pixel art tutorials in order to make my pixel art skills better. The result is this face. The face has been blown up 2x to make it easier to see. The idea is to do some pixel art tutorials every week this year. When I am learning something I like to do it by doing a small project. So my pixel art learning project is going to be to update my bleten game. Bleten was a game I made for a 48 hour game dev competition(using python and pygame to code of course). Due to lack of time(and poor pixel-fu) most of the pixel artwork in it is pretty average. Including some stick-sheep.

Also game related, I am working on a milkshape character animation library written in C. I have converted it from some C++ code I had lying around. The conversion from C++ to C code is in order to make it easier to port, make it easier to use as a library, to simplify it, and to reduce code size. I will be wrapping it up with python bindings once done. It should weigh in at 10-20KB once finished.

The current C++ cal3d library I use for character animation I am having a few problems with. Firstly when I update milkshape the exporter needs to be recompiled. Also I need to export models, I can not read directly from the milkshape files. Another problem is that the cal3d library in the last few versions has been using more template features. Which are anoying to use from python. Finally it is quite large. All up(including the python bindings) cal3d weighs in at about 500KB. This is about 1/10th the total size of a 5MB game. 5MB being the magic size small enough for modem users. Another reason for the change is so that I can optimize the character animation code for my needs. Cal3d is quite large, and I have found it hard to optimize. I want to be able to cache frames to speed up playback, as well as to change data types to suit the models I use. I also want to support some features that cal3d does not. Like scaling in animations, and only animating certain bones. Eg only animating the hand with a hand animation.

I am going to wrap the C animation library up with the ctypes wrapper. Hopefully pyopengl will finish its transition to ctypes as well, so that I can save some more space with the reduced size of the pyopengl wrappers.

Pygame patches have also been going in. Someone asked about when a new release would come out... maybe in a couple of months. The only feature I want to add is per pixel collision detection. Also need to expose some new SDL functionality from the last SDL release. There have been enough bug fixes to make it worthwhile to make a release.

For web related things, I have been working on a few sites in php for work. A new version of the antipoda design site(unreleased yet), as well as some improvments to an existing site www.nicholsoncartoons.com.au I have also been doing some more things on www.pretendpaper.com using python.

Dane has mostly completed a new look for the site. Very similar to the old site, but lighter with less dark lines. It makes pretendpaper somewhat cleaner. The new design should come online in the next month I reckon. Hopefully at the same time I finish a new set of changes.

These new changes include new edit screens. I have just started changing the edit screens around. During so I ran into problems with the session implementation that I use. I need to change the session code around to support multiple readers and writers better. The session bugs pretty much stopped me in my tracks.
*Update:* I hacked on the session code, adding locks for each session rather than for the whole lot. The session code was designed for one request per session owner, not multiple ones. I also made it a memory based session that stores its data occasionally to disk(and on shutdown).